Test and debug with Codex

Testing and debugging with Codex works when tests, not the model, decide what “done” means: AGENTS.md names the test commands, diagnosis runs read-only, every fix starts with a test that fails for the diagnosed reason, and a CI check outside the agent’s reach fails any fix that also edits its tests.

This hub is for developers who let Codex write tests and fixes, and tech leads who decide what those pull requests must prove. The familiar failure: every test passes and the bug ships anyway, because the new tests asserted mocks or an assertion was quietly loosened. Commands checked against Codex CLI 0.157.1.

Which Codex test page answers your question?

Your question	Page
Something is broken: how do I find the cause and prove the fix?	Debugging and bug hunting with Codex
How do I close coverage gaps with tests that catch real bugs?	Test generation and execution with Codex
How strong must a suite be before I trust unread tests?	How strong is your oracle?
How do I stop the agent from editing the checks that judge it?	Protecting the oracle
Codex itself misbehaves (stuck approvals, sandbox errors)	Recover when Codex goes wrong

Tool-neutral practice: testing workflows for every tool.

In what order should you set up testing with Codex?

Each step closes one way a green result can lie.

1. Write the test commands into AGENTS.md. Name three test commands and one rule, as below. Since 0.150.0 an untrusted project does not supply its AGENTS.md, so trust the repository first. More in writing AGENTS.md for Codex.

   ## Testing
   - One file: `npx vitest run path/to/file.test.ts`
   - Tests related to a change: `npx vitest related --run <changed files>`
   - Full suite: `npm test`
   - Never edit a test to make it pass; if one looks wrong, stop and say why.

2. Diagnose read-only. For a bug, start with codex -c default_permissions=":read-only", so nothing changes until Codex names one cause as file, line and condition. Permission profiles are the control OpenAI prefers (beta, Codex CLI 0.138.0 and later); --sandbox read-only is the legacy equivalent. Use one or the other, never both: OpenAI says the two systems do not compose. /plan makes Codex answer with a plan, but only the profile enforces read-only.

3. Prove before fixing. A new test must fail for the diagnosed reason before any fix is written; if it cannot, the diagnosis is wrong. Fix in its own worktree (codex --worktree, or /worktree in a session) so a failed attempt costs nothing.

4. Feed results back automatically. Codex hooks (stable in 0.157.1) include PostToolUse and Stop, so a hook can run related tests after an edit or the fast suite at turn end. Hooks need persisted trust: approve them once with /hooks; untrusted hooks do not run (only --dangerously-bypass-hook-trust, meant for vetted automation, skips this). Registration and a stop gate for failing tests: agent hooks.

5. Lock the tests outside the session. “Do not edit the test” in a prompt is a request, not a control. The lock is a CI check that fails a fix branch changing tests or snapshots without owner approval, plus test paths in CODEOWNERS. Full setup: protecting the oracle.

Copy-paste prompt: triage a CI failure without inviting a patch

CI fails on this branch. The failing job output is in ci-failure.log.
Stay read-only. Read the failing test and every file in the stack trace.
Tell me whether this is (a) a bug in the code, (b) a wrong or flaky test,
or (c) an environment difference between CI and local, and give the one
piece of evidence that decides it. Then propose a new test that fails for
exactly the reason you name. Do not write the fix.

To run this prompt unattended with codex exec and a read-only profile, see non-interactive Codex.

How do you know Codex’s tests and fixes are right without reading every line?

Read the evidence; open the diff only where it points.

Evidence	What it proves	Owner
Test names approved before generation	The suite covers the behaviours you care about	Developer
The new test ran red, for the stated reason, before the fix	The test detects the bug it claims to	Developer
Test files and snapshots unchanged in the fix’s diff	Nobody passed the suite by weakening it	CI check on test paths
Mutation survivors listed and triaged (mutation check)	The suite catches injected bugs, not only today’s code	Reviewer
Full suite, types and lint green in CI	Existing behaviour still holds	CI

What a pull request attaches: the evidence bundle. Why evidence beats the diff: reading evidence instead of code, practised in test-driven fixing.

What goes wrong when Codex tests and debugs its own work?

Symptom	Cause	Recovery
Suite green, bug in production	Tests assert mocks, not behaviour	Add an integration test at the seam; run the mutation check
A fix passes by changing an assertion or a snapshot	Nothing outside the session guarded the test files	Revert the test change; add the CI test-path check and CODEOWNERS entries
Codex ignores your test commands	The project is untrusted, so its AGENTS.md was not loaded (0.150.0 and later)	Trust the repository; ask Codex which instructions loaded
Codex edits files during diagnosis	The session ran with a writable sandbox	Restart with the :read-only profile; git restore what it touched or discard the worktree
The new test fails differently than the diagnosis predicts	Two plausible causes were never separated	Treat the diagnosis as disproved; ask for evidence ruling each cause out

Where to go next with testing and debugging in Codex

Before this stage: building with Codex. After it: review and ship with Codex. Helpers: testing and debugging skills. The same stage in other agents: Claude Code and Cursor.