# How strong is your oracle? Trusting tests you did not read

URL: https://developertoolkit.ai/en/workflows/test/oracle-strength/
Description: Measure whether a test suite catches bugs: mutation score, behaviour coverage, test provenance and flake rate, and the bar a loop must clear to run unattended.

Oracle strength is how likely a test suite is to fail when the code under test is wrong. It is measured on the code a change touches, with four numbers: mutation score, behaviour coverage of the acceptance criteria, test provenance, and flake rate. A loop runs unattended only after all four clear a bar the team set in advance.

An agent opens a pull request for the invoice export. It adds 38 tests, line coverage on the module is 96%, and CI is green. You flip one `>=` to `>` in the date-range filter by hand and run the suite again. It is still green. Nobody read those 38 tests, and now you know they do not check the boundary the ticket was about. This page is for the developer who approves agent pull requests on evidence, and for the tech lead who decides which loops may run overnight.

## What you get from measuring oracle strength

- **Four metric definitions** (mutation score, behaviour coverage, provenance, flake rate) with the exact command that produces each one.
- **A bar per lane** (interactive, background, overnight) that a loop must clear before it runs with less supervision.
- **A CI job** that mutation-tests only the files a pull request changed and fails below your threshold.
- **Four copy-paste prompts** that audit an oracle, kill surviving mutants without touching production code, add negative controls, and hunt flaky tests.
- **The tool differences** for running the audit in Claude Code, Codex, and Cursor.

## Why is a green suite not proof?

A test is an oracle only if it can fail. Green tells you that the checks ran and passed; it says nothing about whether they would have failed against a wrong implementation. Agent-written suites make the gap wider for three reasons: the same agent often writes the code and the tests in one pull request, it optimizes for green because green is the stop condition, and it writes many tests quickly, so volume looks like rigour.

Line coverage does not close the gap. This test executes every line of `inRange` and asserts nothing about the result:

```ts
// 100% line coverage, zero oracle strength
it('filters invoices by date range', () => {
  const result = inRange(invoices, '2026-01-01', '2026-01-31');
  expect(result).toBeDefined();
});
```

Any implementation that returns an array passes it. Coverage counts lines that ran; oracle strength counts behaviour that was checked. The four measures below are how you tell the two apart without reading every test.

This is the control that the DORA research team names. The 2025 DORA report found "a positive relationship between AI adoption on both software delivery throughput and product performance", and that "AI adoption does continue to have a negative relationship with software delivery stability". Its explanation: "Without robust control systems, like strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability" (Google Cloud blog, Nathen Harvey and Derek DeBellis, 2025-09-23).

## What are the four measures of oracle strength?

| Measure | Definition | Produced by | What a weak value hides |
| ------- | ---------- | ----------- | ----------------------- |
| **Mutation score on changed code** | Detected mutants (killed or timed out) divided by valid mutants (detected, survived, or not covered), for the files the change touched | Stryker (JS/TS), mutmut (Python), cargo-mutants (Rust), PIT (JVM) | Assertions that never compare the result, missing boundary cases, dead branches |
| **Behaviour coverage** | Acceptance criteria with a named check that fails when the criterion is violated, divided by all acceptance criteria | The criterion-to-check map in the pull request, plus one recorded negative control per check | Behaviour nobody specified as a check; tests that cover code, not requirements |
| **Test provenance** | For each decisive check: did it exist before the run, who approved it, and did this change edit it | `git log` and `git diff` on the oracle paths | An agent grading its own homework: tests written or loosened in the same change |
| **Flake rate** | Gating tests that both passed and failed on the same commit during the sweep window, divided by gating tests | Repeated runs with retries off | Checks that go red at random, which trains everyone, human and agent, to rerun instead of read |

Line and branch coverage stay as a floor. Keep a threshold so that untested code shows up, but never read a high number as strength.

### Measure mutation score on the code the change touched

A mutation tool makes small deliberate bugs (a flipped comparison, a removed call, a constant changed) and runs your tests against each one. A mutant that makes a test fail is **killed**; one that passes every test **survived**. Stryker computes the score as detected over valid mutants, where detected is killed plus timed out, and valid adds survived and not-covered mutants; mutants that fail to compile are excluded (checked in `mutation-testing-metrics`, bundled with `@stryker-mutator/core` 10.0.0, 2026-09-26).

Score the changed code, not the whole repository. A whole-repository score dilutes a weak new module with years of well-tested code, and a full run is too slow to gate every pull request. Stryker's `--mutate` option takes a comma-separated list of files, and even line ranges (`src/index.js:1:3-1:5`), so you can mutate exactly what the pull request changed:

```jsonc
// stryker.config.json (Stryker 10.0.0)
{
  "testRunner": "vitest",
  "coverageAnalysis": "perTest",
  "reporters": ["clear-text", "progress", "json"],
  "thresholds": { "high": 80, "low": 60, "break": 60 },
  "incremental": true
}
```

`thresholds.break` is the gate: when the final score falls below it, Stryker logs the failure and sets exit code 1. With `break` unset (its default is `null`), Stryker never fails the build, whatever the score. The `high` and `low` values above are Stryker's own defaults for colouring the report, not a research finding; the break value is your team's policy. `incremental` stores results in `reports/stryker-incremental.json` and reuses them on the next run, and the `json` reporter writes `reports/mutation/mutation.json`, which your evidence bundle can read. The runner plugin is a separate package: `npm i -D @stryker-mutator/core @stryker-mutator/vitest-runner`.

The equivalents in other stacks, checked on 2026-09-26:

- **Python:** mutmut 3.8.0. Set `source_paths` (and optionally `only_mutate`) under `[tool.mutmut]` in `pyproject.toml`, run `mutmut run`, inspect with `mutmut results` and `mutmut show <name>`, and write CI numbers with `mutmut export-cicd-stats` (to `mutants/mutmut-cicd-stats.json`).
- **Rust:** cargo-mutants 27.1.0. `git diff origin/main... > pr.diff && cargo mutants --in-diff pr.diff` tests only mutants that overlap the diff. Its own documentation warns that a diff which changes only test code runs no mutants, so a pull request that weakens tests needs the provenance check below.
- **JVM:** PIT (`pitest`) is the established tool; use its Maven or Gradle plugin.

Not every survivor is a gap. An **equivalent mutant** changes the code without changing behaviour (for example, a mutated log message nobody asserts on). Triage each survivor as one of three things: a missing test, an equivalent mutant, or behaviour you decide not to pin. Record the decision; an untriaged survivor counts against the loop.

### Measure behaviour coverage against the acceptance criteria

Mutation score tells you whether the tests notice change. Behaviour coverage tells you whether they check the right things. Start from the ticket's acceptance criteria (see [executable acceptance criteria](/en/workflows/plan/acceptance-criteria/)) and require one line per criterion in the pull request: the criterion, the check that proves it, and a **negative control**, which is the result of breaking that behaviour on purpose and watching the named check go red.

```text
AC-1 reversed range returns 400  -> tests/invoices/export.test.ts "rejects reversed range"
     negative control: swapped the guard to `from < to`; check failed as expected
AC-2 amounts in invoice currency -> MISSING
```

A negative control turns "a test exists" into "this test fails when the behaviour breaks". It is a targeted mutation that you choose, where the mutation tool chooses at random. `MISSING` is an acceptable answer; it moves the criterion to a human, and it is better than a check the agent invented to fill the row.

### Check the provenance of every decisive test

A check proves the most when it existed before the agent started and the agent could not edit it. Two commands answer the provenance questions for a pull request:

```bash
# Terminal or CI: which oracle files did this change touch?
git diff --name-only origin/main...HEAD -- \
  '*.test.*' '*.spec.*' '*__snapshots__*' '.github/workflows/*' \
  'tsconfig*.json' 'vitest.config.*' 'stryker.config.*'

# When was a decisive test first added, and by whom?
git log --diff-filter=A --format='%h %an %ad' --date=short -- tests/invoices/export.test.ts
```

Classify each decisive check as **pre-existing** (added before the run started), **human-approved** (written as its own slice and approved before implementation began), or **same-change** (written or edited in this pull request). Same-change checks can support the evidence, but they do not count toward the bar for background or overnight runs. Enforcing the paths the agent may not edit, through hooks, `CODEOWNERS` and CI-owned checks, is covered in [protecting the oracle](/en/workflows/test/protect-the-oracle/).

### Measure flake rate with retries off

A flaky test weakens the oracle in both directions: it fails good changes, and it teaches people and agents that red means "run it again". Measure it on a schedule, not in the pull request, and with retries disabled so that nothing is hidden:

```bash
# Nightly, from the repository root
npx playwright test --repeat-each 20 --retries 0 --reporter=json > flake-e2e.json
npx vitest run --repeats 10 --reporter=json --outputFile=flake-unit.json
```

Any test that fails in a repeated run on a commit where it also passes is flaky. Quarantine it out of the gating set for every loop that depends on it, fix or delete it, and restore it only after it passes a sweep. If you keep retries in pull-request CI, run Playwright with `--fail-on-flaky-tests` so that a test that passes on retry still fails the run (flags checked in `@playwright/test` 1.63.0 and Vitest 5.0.2, 2026-09-26).

## What bar must a loop clear before it runs unattended?

A **loop** is a repeatable class of change with its own trigger, oracle and stop condition, as defined in [the one map](/en/ladder/one-map/). The bar belongs to the loop, not to the repository: dependency bumps can clear it while checkout feature work does not. The lanes below match those in [shaping a backlog for agents](/en/teams/agent-ready-backlog/).

| Measure | Interactive | Background | Overnight (unattended) |
| ------- | ----------- | ---------- | ---------------------- |
| Mutation score on changed code | Reported | At or above the team's break threshold | At or above the break threshold, and every survivor triaged |
| Behaviour coverage | Every criterion has a check or is marked `MISSING` | 100% mapped, each with a negative control | 100% mapped, each with a negative control |
| Provenance | Oracle files touched are read by a human | Decisive checks pre-existing or human-approved; oracle edits escalate | Decisive checks pre-existing; zero oracle files touched, enforced outside the prompt |
| Flake rate | Known | Zero flaky tests in the loop's gating set | Zero flaky results across the sweep window, retries off |
| Line and branch coverage | Floor | Floor | Floor |

The thresholds are policy you set and write down, not benchmarks. Start with Stryker's default low band (60) as the background break value and its high band (80) for overnight, then move them on your own data: an escaped defect in a loop is a reason to raise the bar or demote the loop, and a long run of clean pull requests with no survivors is a reason to keep it. Organisations that run agents at volume keep the oracle outside the agent's reach and bound the run. Stripe's agents run against "over three million" existing tests and stop after "at most two rounds of CI" (Stripe engineering blog, Alistair Gray, February 2026).

## Run an oracle audit on one loop

The audit takes an afternoon for one loop. Run it before you promote a loop, and again after any escaped defect.

1. **Name the loop and its oracle paths.** Write down the trigger, the stop condition, and the test, snapshot, CI and config globs that make up its oracle. Put the globs in `CODEOWNERS` so that any change to them needs a named reviewer.

2. **Map the acceptance criteria to checks.** For the last five pull requests in the loop, list every criterion and its check. Mark gaps `MISSING`. Use the negative-control prompt below to record one control per check.

3. **Run a baseline mutation score.** Run the mutation tool on the loop's modules with `--incremental` so later runs are fast. Triage every survivor as missing test, equivalent, or accepted.

4. **Close the gaps in a test-only task.** Give the agent the survivors and the `MISSING` criteria, forbid edits to production code, and have a human approve the new tests. They become pre-existing checks for every later run.

5. **Turn on the per-change gate.** Add the CI job below so every pull request in the loop mutation-tests its own changed files and fails below the break threshold.

6. **Start the nightly flake sweep.** Run the repeat commands on a schedule, quarantine anything flaky, and keep the loop out of the overnight lane until a full window passes clean.

7. **Record the result in the evidence bundle.** Each pull request carries the mutation score, the criterion map, the list of oracle files touched, and the flake status, in the format in [the evidence bundle](/en/workflows/ship/evidence-bundle/).

This job mutation-tests only the production files a pull request changed. It relies on `thresholds.break` in `stryker.config.json` for the pass or fail decision:

```yaml
# .github/workflows/oracle-strength.yml
name: oracle-strength
on: pull_request
permissions:
  contents: read
jobs:
  mutation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
          fetch-depth: 0
      - uses: actions/setup-node@v7
        with:
          node-version-file: .node-version
      - run: npm ci
      - name: Mutation-test the files this pull request changed
        env:
          BASE: ${{ github.base_ref }}
        run: |
          FILES=$(git diff --name-only --diff-filter=AM "origin/$BASE...HEAD" -- 'src/*.ts' \
            | grep -vE '\.(test|spec)\.ts$' | paste -sd, -)
          if [ -z "$FILES" ]; then echo "No production files changed"; exit 0; fi
          npx stryker run --mutate "$FILES"
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: mutation-report
          path: reports/mutation/
```

In a Git pathspec, `*` also matches `/`, so `'src/*.ts'` selects TypeScript files at any depth under `src/`.

## Copy-paste prompts for measuring your oracle

**Copy-paste prompt: audit the oracle of this pull request**

```text
Audit the test oracle of the current branch against origin/main. Do not edit
any file.

1. List every file changed under tests, snapshots, .github/workflows, and any
   lint, tsconfig, vitest or stryker config. For each, say what changed and
   whether it makes any check easier to pass.
2. Read the acceptance criteria in the linked issue. For each criterion, name
   the test file and test that proves it, or write MISSING. For each named
   test, say whether it existed on origin/main (use git log).
3. Run `npx stryker run --mutate <the changed production files, comma-separated>`
   and report the mutation score and every surviving mutant with file, line,
   mutator and replacement.
4. Classify each survivor: missing test, equivalent mutant, or unclear.

Output a table per section and one final line: PASS if every criterion has a
pre-existing or human-approved check and no oracle file weakens a check,
otherwise FAIL with the reasons.
```

**Copy-paste prompt: kill surviving mutants without touching production code**

```text
Read reports/mutation/mutation.json. For every mutant with status "Survived"
in src/invoices/, write or strengthen a test that fails against the mutant and
passes against the current code.

Rules:
- You may edit only files under tests/. Do not change anything under src/.
- Assert on observable behaviour (return values, HTTP status and body, emitted
  events), never on private functions or call counts.
- If a mutant cannot change observable behaviour, do not write a test. List it
  as EQUIVALENT with a one-line reason.
- After each new test, rerun `npx stryker run --mutate <that source file>`
  and confirm the mutant is now killed.

Finish with: mutants killed, mutants marked EQUIVALENT, and the new score.
```

**Copy-paste prompt: record a negative control for each acceptance check**

```text
For each acceptance criterion in the issue, find the check that is supposed to
prove it. Then, one criterion at a time:

1. Make the smallest change to production code that violates the criterion.
2. Run only the named check and record whether it failed.
3. Revert the change with `git checkout -- <file>` and confirm the check passes
   again.

Never commit the violating change. Output one line per criterion:
criterion -> check -> negative control: failed as expected / STILL PASSED.
A check that STILL PASSED does not prove its criterion; say what it misses.
```

**Copy-paste prompt: find and quarantine flaky tests**

```text
Run `npx vitest run --repeats 10 --reporter=json --outputFile=flake-unit.json`
and `npx playwright test --repeat-each 20 --retries 0 --reporter=json > flake-e2e.json`
on the current commit. Do not change any test yet.

List every test that both passed and failed. For each one, read the test and
the code it exercises and name the likely cause: shared state, timing or
waits, test order, network, randomness, or date and time. Propose the fix, but
do not apply it. Do not add retries, longer timeouts or skips as a fix.
```

## How do you run the audit in Claude Code, Codex, and Cursor?

The measures, commands and prompts are identical in all three tools, because the mutation and test tools do the measuring. What differs is how you run the audit headless, how you keep the agent from editing the oracle while it works, and where a test-only improvement task runs. Commands were checked against Claude Code 2.1.283 and Codex CLI 0.157.1 on 2026-09-26; Cursor features were checked on cursor.com on 2026-08-28.

Save the first prompt above, "audit the oracle of this pull request", as `prompts/oracle-audit.md`; the headless commands below read it from there. Step 2 of that prompt reads the linked issue, so a headless run needs either `gh` access (the Claude Code allow-list below includes `gh issue view`) or the acceptance criteria pasted into the prompt file.

**Claude Code**

    **Headless audit in CI.** `-p` sessions start in Manual permission mode (auto mode on third-party providers or with telemetry off, from v2.1.285), so grant only what the audit needs. This allows reads, Git history, Stryker and reading the linked issue, and nothing that edits files:

    ```bash
    # CI or terminal, from the repository root. In CI on pull request code, also pass
    # --bare --setting-sources "" --strict-mcp-config (and set ANTHROPIC_API_KEY), so the
    # branch's hooks, settings and .mcp.json do not load
    claude -p "$(cat prompts/oracle-audit.md)" \
      --allowedTools "Read,Grep,Glob,Bash(git diff:*),Bash(git log:*),Bash(npx stryker run:*),Bash(gh issue view:*)" \
      --max-budget-usd 3 --output-format json > oracle-audit.json
    ```

    Add `--json-schema` with a schema for the PASS or FAIL table when a later CI step parses the result. Stryker runs the pull request's tests in a process that inherits the API key, so in CI run this job only for pull requests from the repository itself, never on `pull_request_target`.

    **Test-only improvement.** In an interactive session, set a completion condition with `/goal`, for example `/goal every survivor in reports/mutation/mutation.json under src/invoices is killed or listed as EQUIVALENT, and no file under src/ is modified`. Claude keeps working until the condition is met, a model judges it impossible, or an error clears the goal. Deny edits to `src/**` in your permission settings rather than trusting the prompt, and check `git diff --name-only -- src/` is empty before you accept the result.

    **Mutation skills.** Trail of Bits publishes a `mutation-testing` plugin (1.9.2) that sets up mewt or muton campaigns and analyzes surviving mutants: `claude plugin marketplace add trailofbits/skills`, then `claude plugin install mutation-testing@trailofbits`.

**Codex**

    **Headless audit in CI.** `codex exec` runs non-interactively. A read-only sandbox cannot run Stryker, which writes a sandbox directory and reports, so use `workspace-write` and have a later CI step fail the job if anything outside `reports/` changed:

    ```bash
    # CI or terminal, from the repository root (Codex CLI 0.157.1)
    codex exec --sandbox workspace-write --output-schema oracle-audit.schema.json \
      -o oracle-audit.json "$(cat prompts/oracle-audit.md)"
    ```

    `--output-schema` constrains the final message to your JSON Schema, and `-o` writes that message to a file. Codex has no flag that skips `AGENTS.md`, so in CI the pull request's own `AGENTS.md` steers this run, and Stryker runs its tests with the API key in the environment. Run it only for pull requests from the repository itself, and treat the diff check after it, not the agent's report, as the gate.

    **Test-only improvement.** Use `/goal` with the same completion condition as in the Claude Code tab. Do not rely on the prompt to keep the agent out of `src/`: fail the task if `git diff --name-only -- src/` is not empty, and see [protecting the oracle](/en/workflows/test/protect-the-oracle/) for enforcement that runs before the edit.

    **Mutation skills.** The same Trail of Bits marketplace installs in Codex: `codex plugin marketplace add trailofbits/skills`, then `codex plugin add mutation-testing@trailofbits`.

**Cursor**

    **Audit in the editor.** Paste the audit prompt into the agent with the pull request branch checked out. Start in Plan Mode so that it reports before it touches anything.

    **Test-only improvement.** `/goal` (rolling out since 2026-08-19, and still so on 2026-10-12) gives the agent a long-lived objective; use the same completion condition as in the other tabs, and reject the result if `git diff --name-only -- src/` is not empty.

    **Nightly flake sweep.** A Scheduled Automation can start a Cloud Agent that runs the repeat commands and opens an issue listing flaky tests. Keep that agent's instructions to "report, do not fix", so that the sweep never edits the tests it measures. Setup is on [Cursor Cloud Agents and Automations](/en/cursor-ide/automate/cloud-agents-automations/).

## Who signs off on oracle strength?

Nobody reads every test, so the measurement itself must be checkable:

- **CI computes the numbers.** The mutation score, the list of oracle files touched, and the flake status come from tool output attached to the pull request, never from the agent's summary of it.
- **A human approves every new decisive check** before it counts as pre-existing. That approval is the one place where a person reads test code on purpose.
- **The tech lead owns the thresholds and the oracle globs**, reviews the survivor triage for promoted loops, and demotes a loop when a defect escapes it. The promotion rules are in [helping the team stop reading every diff](/en/teams/trust-transfer/).
- **Every escaped defect becomes a check.** Write the test that would have caught it, confirm it with a negative control, and add it to the oracle before the loop runs unattended again.

## What breaks when you measure oracle strength?

**The agent kills mutants with change-detector tests.** It asserts on call counts, private helpers or exact log strings, the score rises, and the tests break on every refactor while catching nothing new. Recovery: require assertions on observable behaviour (the second prompt does), and reject any new test whose only failure mode is a harmless refactor.

**The score is gamed through scope.** The `mutate` list shrinks, files move into an ignore pattern, or `thresholds.break` drops to make CI pass. Recovery: treat the Stryker config as an oracle file with a code owner, and compute the mutated file list in CI from the diff, as the workflow above does, not from a list the pull request can edit.

**Mutation runs are too slow to gate on.** A whole-repository run on every pull request takes too long, so the team turns it off. Recovery: mutate only changed files per pull request, use `--incremental`, and run the full score weekly on a schedule.

**Timeouts count as kills.** Stryker counts a timed-out mutant as detected, which is correct for an infinite loop but can hide a slow, flaky test. Recovery: if many mutants time out in one file, look for tests with long waits before trusting the score.

**Retries hide the flake rate.** A retry turns a flaky failure into a pass and the rate reads zero. Recovery: measure flakiness only in the nightly sweep with retries off, and use `--fail-on-flaky-tests` wherever retries stay on.

**Same-change tests are counted as proof.** The agent writes the code and the tests that bless it, and every number looks healthy. Recovery: the provenance classification decides what counts; same-change checks never clear the background or overnight bar on their own.

## Where to go next with oracle strength

- [Protecting the oracle](/en/workflows/test/protect-the-oracle/) — Hooks, CODEOWNERS and CI-owned checks that keep the agent out of the tests.
- [The evidence bundle](/en/workflows/ship/evidence-bundle/) — Where the mutation score, criterion map and oracle changes go in every pull request.
- [Property-based testing](/en/workflows/test/property-based-testing/) — Invariants from the spec: an oracle an agent cannot overfit by example.
- [Characterization tests](/en/workflows/test/characterization-tests/) — Build an oracle for legacy code before an agent touches it.
- [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) — Where oracle strength sits in what a reviewer reads instead of the diff.
- [Shaping a backlog for agents](/en/teams/agent-ready-backlog/) — The eligibility test and lanes that this page's bar plugs into.
