Test generation and execution with Codex

Test generation with Codex closes coverage gaps safely when each module gets its own Git worktree lane, the lane may add tests but never change source code, and the result is judged by evidence rather than by reading it: a tests-only diff gate, a per-file coverage delta, a mutation score, and a codex review pass before merge.

Your coverage report says 34%, the release checklist says 80%, and 200 uncovered functions sit across 40 files. You could ask Codex to “add tests until coverage hits 80%”, and it would, with tests that assert mocks were called and pass whatever the code does. This page is for developers who hand test writing to Codex, and for tech leads who decide what a test-generation pull request must prove before it merges. Commands were checked against Codex CLI 0.157.1 on 2026-09-26.

What you’ll walk away with from Codex test generation

• A lane layout that runs several Codex test tasks in parallel worktrees without them touching each other or your checkout.
• Three copy-paste prompts: gap analysis, convention-matched test generation, and an adversarial audit of the generated tests.
• A gate script that rejects any lane that edited source code, and a coverage-delta check per file.
• A mutation check with StrykerJS that shows whether the new tests catch bugs, not only execute lines.
• A weekly test-health run you can schedule, and a failure-modes table with the recovery for each.

If you are new to worktrees in Codex, read isolated Codex tasks with Git worktrees first. Debugging a failing test is a different job, covered in debugging and bug hunting with Codex.

Who checks what when Codex writes the tests?

Coverage percentage is the number everyone asks for and the weakest evidence you have. A test that calls a function and asserts nothing raises line coverage as much as a good one. Split the work so each artifact has a check that is not Codex’s own opinion.

Artifact	Produced by	Verified by	Signed off by
Priority list of files to test	Codex, read-only	You read 10 lines, not the code	Developer
Planned test names per lane	Codex	You read the list before any test is written	Developer
Generated test files	Codex, in a worktree lane	Tests-only gate; five green runs in a row	Gate script
Coverage gain	The test run	Per-file delta against the lane’s own baseline	Gate script
Bug-catching strength	StrykerJS	Mutation score and the list of surviving mutants	Reviewer reads the survivors
Test quality	A second Codex pass	codex review plus the audit prompt	Reviewer reads the findings
Merge	CI	Full suite, types, lint on a clean machine	Code owner, on the evidence

Why a green suite is not proof on its own, and how strong the suite must be before you stop reading it, is covered in how strong is your oracle?

Prepare the repository before Codex writes a single test

1. Name the test commands in AGENTS.md. Codex reads AGENTS.md before it starts, so it runs your commands instead of guessing. Since Codex CLI 0.150.0 an untrusted project does not supply its project AGENTS.md, so trust the repository first: start codex in the repository root and accept the Trust this folder? prompt.

   ## Tests
   - One file: `npx vitest run tests/unit/services/orders/createOrder.test.ts`
   - Full suite: `npx vitest run`
   - Coverage: `npx vitest run --coverage --coverage.reporter=json-summary --coverage.reporter=text`
   - Tests live in tests/unit/ and tests/integration/, mirroring src/.
   - Never edit a test to make it pass, and never change src/ in a test-writing task.

   More on the file in writing AGENTS.md for Codex.

2. Record a coverage report on the base commit. Run the coverage command from the list above yourself. The json-summary reporter writes coverage/coverage-summary.json, which the gap-analysis prompt reads and the gate compares against later. The coverage flag needs @vitest/coverage-v8 installed; if your repository lacks it, do step 4 first.

3. Optionally install a test-first skill. The tdd skill from mattpocock/skills makes Codex plan behaviours before writing tests. Install it at project scope with npx skills add mattpocock/skills --skill tdd codebase-design -a codex -y. codebase-design is included because tdd calls it for its vocabulary, and installing tdd alone leaves that reference dangling. The options and traps are in testing and debugging skills.

4. Set up mutation testing on the base branch. The mutation check later on needs StrykerJS in every lane, and the coverage flag needs @vitest/coverage-v8 installed, so both go into the same base commit. Install it, configure it and ignore its output in one commit, and land that commit on your default branch (through a pull request if the branch is protected) before you create any lane:

   npm install -D @vitest/coverage-v8 @stryker-mutator/core @stryker-mutator/vitest-runner
   echo '{ "testRunner": "vitest" }' > stryker.config.json
   printf 'reports/\n.stryker-tmp/\n' >> .gitignore
   git add package.json package-lock.json stryker.config.json .gitignore
   git commit -m "Add StrykerJS and the Vitest coverage provider"

   Stryker writes its HTML report to reports/mutation/ and its sandbox copies to .stryker-tmp/, so both must be ignored, or the tests-only gate reports them as stray files. The lanes branch from origin/main, so each one inherits this setup and a lane never touches package.json.

Find the coverage gaps worth closing first

Low coverage in a config loader matters less than low branch coverage in the code that takes money. Ask Codex to rank the gaps, and keep it read-only while it does: analysis needs no write access.

Copy-paste prompt: rank coverage gaps by risk

Read coverage/coverage-summary.json and the source files it lists.
Do not modify any file and do not run the test suite.

Rank the 10 files where missing tests carry the most risk. Weigh branch
coverage over line coverage, and weigh code that moves money, changes
persisted state, or enforces permissions over utilities and config.

For each file give: path, line %, branch %, the uncovered functions or
branches by name, and one sentence on what breaks in production if that
path is wrong. Then group the 10 files into at most 4 lanes of related
files that one agent can test without touching another lane's files.
Output a Markdown table followed by the lane list.

Run it interactively, or unattended with a read-only sandbox and no saved session:

codex exec --sandbox read-only --ephemeral -o coverage-gaps.md - < rank-gaps.txt

rank-gaps.txt holds the prompt above. You read coverage-gaps.md, adjust the lanes, and that ten-line table is the only thing you approve at this stage.

Generate tests in parallel worktree lanes

Each lane gets its own worktree and branch, so two agents never edit the same checkout, and a failed lane is deleted without cleanup in yours. git worktree add gives you named branches that later become pull requests; codex --worktree is the interactive alternative when you drive one lane by hand. How many lanes to run at once and in which order to merge them is covered in Codex multi-agent workflows.

1. Create one worktree per lane and install dependencies yourself. A new worktree has no node_modules and no gitignored files. Install from your own terminal, because the workspace sandbox withholds network access from Codex. Then record the lane’s own coverage baseline: the summary keys are absolute paths, so a baseline from your main checkout does not match the lane’s files.

   git fetch origin
   for lane in orders payments auth; do
     git worktree add -b "tests/$lane" "../app-tests-$lane" origin/main
     (cd "../app-tests-$lane" && npm ci \
       && npx vitest run --coverage --coverage.reporter=json-summary >/dev/null \
       && cp coverage/coverage-summary.json "/tmp/coverage-before-$lane.json")
   done

   Replace origin/main with your default branch and the lane names with the lanes from the gap analysis.

2. Approve the planned test names for each lane. Start codex in a lane worktree, paste the generation prompt below, and stop at the list of test names. Reading 15 test names takes a minute; reading 400 lines of test code does not happen. Add any behaviour you care about that is missing.

3. Run the lanes. Either continue each approved session interactively, or save each lane’s prompt with the approved names appended and run the lanes unattended in parallel:

   for lane in orders payments auth; do
     codex exec -C "../app-tests-$lane" -c default_permissions=":workspace" \
       --ephemeral -o "/tmp/lane-$lane.md" - < "prompts/tests-$lane.txt" &
   done
   wait

   default_permissions=":workspace" selects the built-in workspace permission profile, which is beta in 0.157.1; on a Codex version without profiles, use --sandbox workspace-write instead. Do not combine the two: OpenAI says the profile and legacy sandbox systems do not compose. Every flag and output option of codex exec is covered in running Codex non-interactively.

Copy-paste prompt: convention-matched tests for one lane

You are in a Git worktree for one test-writing lane. Work only here.
Target files:
- src/services/orders/createOrder.ts
- src/services/orders/cancelOrder.ts
- src/services/orders/refundOrder.ts

1. Read tests/unit/services/payments/capturePayment.test.ts and
   tests/unit/services/users/createUser.test.ts. List the conventions:
   describe/it nesting, mock setup, fixture helpers in tests/fixtures/,
   assertion style and test naming.
2. List the test names you plan to write, one line each. Cover every
   exported function, every thrown error, every catch branch, and the
   boundaries: zero, empty list, maximum length, expired or already
   cancelled state. Stop here if I am in the session; otherwise continue.
3. Write the tests in tests/unit/services/orders/<functionName>.test.ts
   using exactly those conventions. Mock only true external edges (the
   database client, the payment provider, the email sender). Assert
   observable results: return values, thrown errors, persisted rows and
   calls that ARE the behaviour (an email sent, a refund issued), never
   internal helper calls.
4. Do not modify anything under src/. If a test exposes what looks like
   a bug, keep the test, mark it `it.skip` with a comment starting
   `BUG?:` that says what you expected, and list it in your report.
5. Run each new file with `npx vitest run <file>` until it passes, then
   run the full suite. Do not commit.

Report: files created, test count per file, the full-suite result with
its exit code, and every BUG?: test.

The BUG?: rule matters. When a test fails because the code is wrong, the easy way to green is to change the assertion to match the bug. Forbidding source edits and asking for a flagged, skipped test turns a hidden bug into a line in the report that you triage.

Prove the generated tests catch bugs without reading them

Run these checks in each lane before you open a single test file, and stop at the first failure. The first two are mechanical; the last two measure test strength.

1. Tests-only gate. A test-writing lane that changed source code is rejected, whatever its results:

#!/usr/bin/env bash
# tests-only-gate.sh LANE — run from inside the lane's worktree
set -euo pipefail
lane="$1"
stray=$(git status --porcelain --untracked-files=all | cut -c4- | grep -v -E '^tests/' || true)
[ -z "$stray" ] || { echo "FAIL: lane changed files outside tests/:"; echo "$stray"; exit 1; }

# The new tests must pass five times in a row: a flaky generated test is worse than none
for i in 1 2 3 4 5; do npx vitest run tests/ >/dev/null || { echo "FAIL on run $i"; exit 1; }; done

# Per-file branch-coverage change against this lane's own baseline
npx vitest run --coverage --coverage.reporter=json-summary >/dev/null
node -e '
const a = require(process.argv[1]), b = require("./coverage/coverage-summary.json");
for (const f of Object.keys(b)) {
  if (f === "total") continue;
  const d = b[f].branches.pct - (a[f]?.branches.pct ?? 0);
  if (d !== 0) console.log(d.toFixed(1).padStart(6), f);
}' "/tmp/coverage-before-$lane.json"
grep -rn "BUG?:" tests/ || true

The gate expects coverage/ to be gitignored. A target file missing from the coverage output means the tests never reached it, however many there are. Send the lane back with the uncovered branch named by line.

2. Mutation check. Line coverage shows which lines ran; mutation testing shows whether a wrong line would be noticed. StrykerJS changes the source in small ways (a flipped condition, a removed branch, > for >=) and reruns the tests; a mutant that survives marks behaviour no test pins down. In a scratch run with a single happy-path test, Stryker 10.0.0 reported full line coverage and still found surviving mutants, which is the gap this check exists for.

npx stryker run --mutate "src/services/orders/**/*.ts"

Stryker is already installed and configured in the lane, because the lane branched from the base commit you prepared in step 4 of the setup. Scope --mutate to the lane’s target files: a full-repository run takes far longer and tells you nothing about this lane. The bar to aim for and how to read survivors are in how strong is your oracle?

3. Review pass. Run codex review --uncommitted in the lane for Codex’s built-in review, or type /review in an interactive session and then steer it with the audit prompt below. In 0.157.1, codex review does not accept custom instructions together with --uncommitted or --base (checked 2026-09-26), so standing rules such as “flag tests without a meaningful assertion” belong in AGENTS.md. The same reviewer setup at pull-request scale is covered in code review with Codex.

Copy-paste prompt: adversarial audit of generated tests

Audit the uncommitted test files in tests/ as if you want to prove them
useless. Do not edit any file.

For each test, answer:
1. Would this test still pass if the function under test returned a
   hard-coded value, threw nothing, or skipped its database write?
2. Does it assert only that a mock was called, or that a value is
   truthy or defined?
3. Does a mock return a shape the real dependency never returns? Check
   the real implementation of each mocked module before answering.
4. Does it depend on test order, wall-clock time or shared state?

Then take the surviving mutants from the Stryker output I paste below
and, for each one, name the missing assertion and the test it belongs in.
Output a table: test name, problem (or "ok"), fix in one sentence.

4. Evidence for the pull request. Commit the lane yourself once the gate passes, push the branch and open a pull request that carries the coverage delta, the mutation score with its survivors, the BUG?: list and the audit table. Reviewers read that evidence and open only the tests it flags. What a pull request should attach is defined in the evidence bundle. To stop a later task from weakening these tests, put tests/ under CODEOWNERS and add the CI check described in protecting the oracle.

Keep coverage and test health from sliding back

A suite that was strong in September decays as new code lands without tests. Two controls hold the line.

First, make CI enforce a floor. Vitest’s coverage.thresholds setting in your Vitest config fails the run when coverage drops below the numbers you set; raise them each time a lane merges, never lower them to get a pull request through. How to wire the job into GitHub Actions is in CI/CD with Codex.

Second, schedule a weekly test-health report. Codex automations in the desktop app run a prompt on a schedule; a CI cron job running codex exec does the same from your pipeline. The trade-offs, sandbox choices and failure modes are in scheduled Codex automations.

Commit the base-commit summary from setup step 2 once as coverage-baseline.json, so the first scheduled run has something to compare against.

Copy-paste prompt: weekly test-health report

Weekly test-health check. Do not modify source or test files.
1. Run `npx vitest run` three times. List every test that failed in some
   runs but not all: those are flaky. For each, name the likely cause
   (timer, shared state, ordering, network) with the line that shows it.
2. Run the coverage command from AGENTS.md and compare
   coverage/coverage-summary.json with coverage-baseline.json. The summary
   keys are absolute paths, so match files by their path relative to the
   repository root, not by the full key. List every
   file whose branch coverage dropped by more than 2 points, and the
   commits since last week that touched it (`git log --since=7.days -- <file>`).
3. List every test marked `it.skip` or `it.todo`, oldest first.
Report: test count, flaky tests, coverage drops, skipped tests. Do not
propose fixes unless a test failed in all three runs.

Update coverage-baseline.json by hand, in a reviewed commit, after a lane merges. A job that rewrites its own baseline can hide a regression in the same run that caused it.

When Codex test generation goes wrong

Symptom	Cause	Recovery
Coverage went up, mutation score stayed low	Tests execute code but assert mocks or truthiness	Run the audit prompt; resend the lane with the surviving mutants listed as required assertions
The gate reports files outside tests/	Codex “fixed” code to make a test pass	Discard those changes with git checkout -- src/; rerun the lane with the BUG?: rule; triage the flagged behaviour yourself
A target file shows no coverage change	Tests import a mock or a different module path	Name the uncovered branch by line in a follow-up and ask Codex to run coverage for that file only
Tests pass in the lane, fail after merge	The lane branched from an older base, or two lanes changed a shared fixture	Rebase the lane on the current base and rerun the gate; keep fixtures in one lane or in a separate first change
Mocks drift from reality	Mocks were written from the test’s point of view, not the dependency’s	Ask for mocks built from the real module’s types; add one integration test per seam, as in integration test patterns
Lanes fail on the first command with missing modules	The worktree has no node_modules and the sandbox blocks installs	Run npm ci in the lane from your own terminal, then restart the lane
The suite got slow	Each generated integration test builds its own database	Require shared setup in beforeAll and per-test transactions; see test data management
Codex ignores your test commands	The project is untrusted, so AGENTS.md was not loaded (0.150.0 and later)	Start codex in the repository root and accept Trust this folder?, then ask Codex which instructions it loaded; details in writing AGENTS.md for Codex

Where to go next after generating tests with Codex

Protecting the oracle Lock test paths so a later agent task cannot weaken them

Code review with Codex Run the same reviewer on every pull request

Unit test strategies What to test and how to mock, for any agent

Test and debug with Codex The hub for this stage and its setup order

Using another agent as well? The same job is covered in test generation and TDD with Claude Code and testing patterns in Cursor.