guide

mutation testing on the lines a pull request changed

coverage says a line ran under the tests. mutation testing says whether any test would notice that line breaking. when agents write more of the code, and more of its tests, that is the question a review needs answered, on the lines that changed.

what mutation testing tells a reviewer

a mutation tool changes your code in small ways, such as turning min into max or < into <=, and runs the tests against each change. a mutant the tests catch is a line they pin down. a mutant that survives is a change no test noticed: the code could be wrong there and the suite would stay green.

that matters most for code an ai wrote, because ai-written tests are often circular: they assert that a function returns what it returned, reach every line, and catch nothing. coverage cannot see that. mutation testing can.

what a room shows

• on a changed line, at its end: 1 mutant not caught in amber, or 2 caught, quietly;
• under a line no test pins down: mutant survived: no test noticed this change, what the tool changed, and ask for a test that catches it, which opens a dispatch to the patcher agent with the line selected and the mutant written into the intent, for a person to send;
• for each file: caught 2 of 3 on changed lines · 1 survived, with the tool and the ci run it came from, linked;
• for the whole change, beside the checks: mutation: caught 42 of 45, or no mutation report on this commit.

only mutants on the lines the pull request added are shown. the rest of a report is about code the change did not touch.

read from your ci, never run by us

diff social does not run mutation testing, or anything else from your repository. your ci runs the tool and uploads its report as an artifact; the room reads it as data, when the head commit's checks finish. it reads two formats:

mutmut writes an internal format that changes between versions, so it is not read until it has a public export.

numbers that do not lie

• no report is not a zero. a commit without one says so, and shows no score;
• a score states what it is out of: the mutants that say something about the tests. ones that did not build or errored are not counted, and the room says how many;
• which test caught a mutant is unknown unless the report says. cargo-mutants does not; stryker does when it ran with per-test coverage;
• a status neither tool documents is refused, and so is a cargo-mutants run whose baseline failed: it says nothing about the tests;
• every report names its source: the run, the artifact, the file in it, and its sha-256.

a real run

diff social's own repository runs it on pull requests that carry a mutants label. on one that taught a flag to spot python's Any, cargo-mutants made 10 mutants on the changed lines: 8 caught, 2 that did not build, none missed. the mutants themselves took 73 seconds; the whole job, with a cold cache, 7 minutes on a 2-core runner. the room showed caught 8 of 8 on changed lines · 2 not counted.

producing a report: cargo-mutants

mutate only what the pull request changed, on its head commit (not the merge github makes, so the report's lines are the pull request's), and upload the result under a name with mutants in it:

on:
  pull_request:
    types: [opened, synchronize, reopened, labeled]
    paths: ["**/*.rs"]

jobs:
  mutants:
    if: contains(github.event.pull_request.labels.*.name, 'mutants')
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v7
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          fetch-depth: 0
      - run: git diff "${{ github.event.pull_request.base.sha }}...HEAD" -- '*.rs' > changes.diff
      - run: cargo install cargo-mutants --locked
      # 0: every mutant caught; 2: some missed; 3: some timed out. Anything else says nothing.
      - run: |
          status=0
          cargo mutants --in-place --in-diff changes.diff || status=$?
          case $status in 0|2|3) ;; *) exit $status ;; esac
      - uses: actions/upload-artifact@v7
        with:
          name: mutants
          path: mutants.out/outcomes.json

two more things diff social's own workflow does: it checks that caught, missed, timed out and unviable add up to the mutants found before uploading anything, because a run cut short can report its partial results as if they were whole; and it fails the job on a missed mutant only after the upload, so the room shows what was missed and where. pin the actions by commit in a real workflow.

producing a report: stryker

- run: npx stryker run --reporters json --incremental
- uses: actions/upload-artifact@v7
  if: always()
  with:
    name: stryker-mutation
    path: reports/mutation/mutation.json

what it costs

mutation testing runs your tests once per mutant, so it is the most expensive job in ci. three things keep it affordable: mutate only the changed lines (--in-diff, or stryker's --incremental); run it when a pull request asks for it, with a label, rather than on every push; and reuse the build cache, one mutant at a time. most pull requests change a few dozen lines, and make a few dozen mutants at most.

checked on 2026-10-19 against cargo-mutants 27.1.0 (its ci guide) and the mutation testing report schema.

where it fits

mutation evidence sits beside the rest of what a room knows about a change: the checks, the agents' lanes, the files that usually change with these, and the conversation on the lines. agents propose; people decide, now with the tests' blind spots in view. more on that: ai code review agents that propose, not nitpick.