Testing and verification for agent-written code

Verifying agent-written code means building checks strong enough that a green run can be trusted without a human reading every line.

An agent opens a pull request with 600 changed lines and 40 new tests, all green. Reading it costs an afternoon; merging on green trusts tests written by the agent that had to pass them. This section is for developers and tech leads who want a third option: make the checks worth trusting, then review the evidence instead of the diff.

Which testing page do you need first?

Read top to bottom: a strong suite the agent can edit proves nothing.

Your question	Page	Why it comes here
Do these tests catch bugs at all?	Oracle strength	The bar every other page builds on.
Can the agent change the tests to make them pass?	Protect the oracle	Keeps the decisive checks out of the agent’s reach.
Which tests should the agent write for each layer?	Unit, integration, API, end-to-end	Tests written from acceptance criteria, not from the code.
Does the running app behave, not only the code?	Browser verification for agents	Screenshots and traces as evidence.
Does the agent itself keep behaving across model or prompt changes?	Continuous evals	Regression tests for the agent harness, run in CI.
What about behaviour that examples cannot capture?	Property-based tests, model-graded checks	Invariants the agent cannot game, and rubric judges for copy and visuals.
It works, but will it stay maintainable and safe?	Fitness functions, slop detection, dependency checks, code quality gates	Gates for erosion, plausible-but-wrong code and invented packages.
The code has no tests yet.	Characterization tests	Pin today’s behaviour before an agent refactors it.
Do I need a specialist gate?	Accessibility, performance, security, error injection, mobile, test data	Add when the product needs them.

Start with an oracle audit of one module

Pick the module the agent changes most. The first prompt measures what its tests catch and changes nothing.

Copy-paste prompt: audit how well the tests catch bugs

Audit the tests for src/billing/ and report how well they catch bugs. Do not edit source, test or config files; tool reports and temp directories are fine.

1. Run the repository's mutation testing tool on src/billing/ only
   (Stryker for JS/TS, mutmut for Python, cargo-mutants for Rust). If none is
   configured, say so and propose the minimal config instead of installing it.
2. Report the mutation score and list every surviving mutant as file:line,
   what was changed, and which behaviour a test should have asserted.
3. List tests that assert nothing meaningful: no assertion, assertions on mocks
   only, or snapshots nobody could review.
4. Report flaky tests: run the suite three times and list any test whose result changes.
Finish with a table: behaviour, test that covers it (or "none"), confidence.

If no mutation tool is configured, the setup is npm i -D @stryker-mutator/core (Stryker 10.0.0), pip install mutmut (mutmut 3.8.0) or cargo install cargo-mutants (cargo-mutants 27.1.0); oracle strength covers configuration.

The second prompt writes failing tests from executable acceptance criteria before the implementation.

Copy-paste prompt: failing tests from acceptance criteria first

Read the acceptance criteria in specs/refund-limits.md. Before writing any
implementation code:

1. Write one test per criterion under tests/refunds/, named after the criterion.
   Assert on observable behaviour (return values, persisted state, emitted events),
   not on internal calls.
2. Run them and show that every new test fails for the expected reason.
   A test that passes now is testing nothing: rewrite it.
3. Stop and show me the criterion-to-test mapping and the failing output.

Do not modify existing tests. If an existing test conflicts with a criterion,
report the conflict instead of changing the test.

Save the first prompt as prompts/oracle-audit.md; the headless commands below read it from there. The prompts work in every tool; only running them differs.

Claude Code

Paste the prompt into an interactive claude session, or run it headless. claude -p starts in Manual permission mode (auto mode on third-party providers or with telemetry off, from v2.1.285, where a classifier decides), so add --permission-mode dontAsk to deny Bash calls it cannot ask about on every provider, and allow the test and mutation commands explicitly (here for a JS/TS project): claude -p "$(cat prompts/oracle-audit.md)" --permission-mode dontAsk --allowedTools "Bash(npx stryker *)" "Bash(npm test *)" (flags: 2.1.283; the v2.1.285 change: Claude Code changelog, checked 2026-10-12). For the failing-tests prompt, which writes files, pass --permission-mode acceptEdits instead. To stop the agent editing tests in the second prompt, add deny rules for your test paths as shown in protect the oracle. Tool-specific setup is in testing with Claude Code.

Codex

Paste the prompt into an interactive codex session, or run it non-interactively with codex exec --sandbox workspace-write "$(cat prompts/oracle-audit.md)"; mutation tools write reports, so a read-only sandbox fails. Checked against Codex CLI 0.157.1. Tool-specific setup is in testing with Codex.

Cursor

Paste the prompt into the Agent chat, or run it non-interactively with the Cursor CLI’s print mode (-p, --print), which also runs in GitHub Actions. Cursor has no built-in per-path deny rule for test files; a hook can block those edits (protect the oracle), but CI is the enforcement point: CODEOWNERS on test paths and a required check. Tool-specific setup is in testing patterns in Cursor.

How do you know the tests are trustworthy without reading them?

Four signals replace reading every test.

Signal	Good enough when	Owner
Mutation score on the changed code	Surviving mutants are listed and each one is killed or explained	Developer who runs the agent
Criterion-to-test mapping	Every acceptance criterion has a test with a recorded failing run	Developer; reviewer spot-checks
Oracle protection	Test and CI config changes need a code owner’s approval	Tech lead
Flake rate	Flaky tests are quarantined, not retried until green	Tech lead

These signals go into the pull request’s evidence bundle, which the reviewer signs off; the argument is in evidence, not diffs.

What breaks when agents write their own tests?

The suite is green but bugs still reach production. The tests assert on implementation details or mocks. Recovery: run the audit prompt, then have the agent kill each surviving mutant with a behaviour assertion.

The agent loosened a test to make it pass. A toBe(19.99) became toBeCloseTo(20, 0) inside a “fix”. Recovery: revert the test change and put test paths behind deny rules and CODEOWNERS.

A coverage target became the goal. Agents chasing a percentage add tests that execute lines without checking them. Recovery: replace the coverage target with the criterion-to-test mapping and the mutation score on changed code.

The suite got slow and flaky after a batch of generated tests. Recovery: drop tests that kill no mutant the rest of the suite misses, and quarantine any test whose result changes between identical runs.

Where to go after the testing section

Executable acceptance criteria Criteria precise enough to become failing tests.

Oracle strength Measure what your tests catch.

Evidence bundle Package test results for the reviewer.

Agent PR review Review agent pull requests by evidence.