Debugging and bug hunting with Codex

Debugging with Codex works when every bug follows one loop: reproduce, isolate, diagnose, fix, verify. Codex writes a failing test before it proposes any fix, and that test becomes the proof the fix works. The surface depends on the bug: the CLI for local loops, a worktree for isolated fixes, Codex cloud for environment-specific failures, and an @codex mention for red pull requests.

This page is for developers who use Codex daily and for tech leads who want the team’s bug fixes to arrive with evidence instead of a diff to read line by line. The situation: users report that saving their profile settings shows the success toast, but the change is gone after a refresh. The API returns 200, the logs are clean, and the database holds stale data. You want the cause found, fixed, and proven fixed before the afternoon is over, without reviewing every line Codex touched.

What you get from reproduction-first debugging with Codex

• A five-step loop that works on every Codex surface, with the rule that no fix starts before a failing test exists
• A decision table for choosing the CLI, codex exec, a worktree, Codex cloud or an @codex mention for a given bug
• Seven copy-paste prompts: reproduction, diagnosis, the fix, regression bisecting, production errors, intermittent failures and memory leaks
• An evidence table that tells you whether a fix is right without reading the whole diff
• Recovery steps for the ways Codex debugging sessions stall

The debugging loop: reproduce, isolate, diagnose, fix, verify

The order matters more than the prompt wording. Most bad agent fixes come from skipping step 1: without a reproduction, Codex fixes the bug it imagines, adds a defensive null check, and reports success.

1. Reproduce. Codex turns the report into a test or script that fails on the current code for the reason the user saw. This failing test is the acceptance criterion for everything that follows.
2. Isolate. Narrow the failure to one module, one input or one commit. git bisect run with the failing test does this mechanically for regressions.
3. Diagnose. Codex states the root cause with evidence: file and line, the value that is wrong, and why. Ranked hypotheses beat a single confident guess.
4. Fix. The smallest change that makes the reproduction pass, made in an isolated worktree so it cannot mix with your other work.
5. Verify. The reproduction passes, the full suite passes, type check and lint pass, and the reproduction still fails if you revert the fix. That last check proves the test tests the bug.

Split diagnosis and fixing into separate prompts. When one prompt says “find and fix”, Codex tends to stop at the first plausible cause. When the diagnosis must stand on its own first, you can reject a wrong theory before it turns into code.

Which Codex surface fits which bug?

Codex runs the same agent on several surfaces, and they share ~/.codex/ configuration. Choose by where the bug reproduces and how much of your attention it needs.

The bug	Surface	How to start it
Reproduces on your machine, needs back-and-forth	CLI, interactive	codex in the repository root
You already have the failing output and want a diagnosis with no edits	CLI, headless	npm test 2>&1 | codex exec -c default_permissions=":read-only" "..."
The fix must not touch your uncommitted work	CLI, in a worktree	codex --worktree, or /worktree in a running session
UI bug you can show but not easily describe	CLI with a screenshot	codex -i broken-header.png "..."
Reproduces only in a specific environment, or flaky enough to need several tries	Codex cloud	codex cloud exec --env ENV_ID --attempts 3 "..."
A pull request’s CI run is red	GitHub	Comment @codex with instructions on the pull request
Production error tracked in Sentry	CLI with the Sentry MCP server	codex mcp add sentry --url https://mcp.sentry.dev/mcp

codex exec reads a piped stdin: when you pass both a prompt and piped output, Codex CLI 0.157.1 appends the output as a <stdin> block. The -c default_permissions=":read-only" override selects a built-in permission profile, which is a beta feature in Codex CLI 0.157.1. ENV_ID is a Codex cloud environment ID. To create one, see Codex cloud environments. For working in the Codex desktop app instead of the CLI, see the Codex desktop app guide.

Reproduce the bug before Codex changes any code

Give Codex the steps, the expected result and the actual result, then forbid a fix. A reproduction prompt that allows fixing gets a fix. Start an interactive session with codex in the repository root and paste:

Copy-paste prompt: reproduce first, fix nothing

Bug report: on /settings/profile, changing the display name and clicking Save shows the "Saved" toast, but after a page refresh the old name is back. PUT /api/users/:id/profile returns 200 and the server logs show no errors.

Your job in this step is ONLY to reproduce it:
1. Find the handler for PUT /api/users/:id/profile and the test file that covers it.
2. Write one failing test that performs the PUT, then a GET, and asserts the new display name is returned.
3. Run only that test and show me the failure output.
4. Confirm the failure matches the report (the value is stale), not a setup error such as a missing fixture or a connection failure.

Do not modify any source file outside the test directory. Do not propose a fix yet. If you cannot make the test fail, stop and tell me what you tried and what state you think is missing.

Check one thing in the output before moving on: the failure message. A test that fails with ECONNREFUSED or fixture not found reproduces your environment, not the bug. Send it back with “the test fails for the wrong reason; fix the test setup, not the code”.

When the bug cannot run locally, reproduce from evidence instead: the exact error, the stack trace, the request that triggered it, and the commit where it started. The recovery section covers that path.

Diagnose the root cause with evidence

Keep the diagnosis read-only. You can say so in the prompt, but a permission profile enforces it. If you already have failing output, pipe it straight in:

npx vitest run tests/api/profile.test.ts 2>&1 | codex exec -c default_permissions=":read-only" \
  "Diagnose the failing test in the attached output: name the root cause with file and line, show the evidence, and do not edit files."

In an interactive session, continue from the reproduction with:

Copy-paste prompt: root cause with ranked hypotheses

The failing test you wrote reproduces the bug. Now diagnose it without editing any file.

1. Trace the request from the route handler to the database call. Name every function on the path with file and line.
2. List up to three hypotheses for why the update is lost (for example: an update query that matches zero rows, a transaction that is never committed, a cache that serves the old value, a serializer that drops the field). Rank them.
3. For each hypothesis, run one command or add one temporary log line that confirms or rules it out, and show the result. Remove any temporary logging afterwards.
4. End with the confirmed root cause in two sentences, the exact line responsible, and the smallest change that would fix it.

If no hypothesis is confirmed, say so. Do not guess.

A good answer names a line and shows evidence, for example: “the Drizzle update in src/routes/profile.ts:48 calls .set() with no .where() on the user ID, so it matches zero rows, and the handler returns the pre-update record”. A weak answer says “possibly a caching issue” without a command that proved it. Ask for the evidence before you accept the theory.

For UI bugs, attach a screenshot. -i (--image) works on codex and on codex exec:

codex -i broken-header.png "The dashboard header in this screenshot should show the navigation items in one row; they stack vertically below 1280px. Find the CSS rule responsible and explain it before changing anything."

Isolate a regression with git bisect

When a feature worked last week and does not now, stop reading commits by hand. The failing reproduction test turns git bisect into a mechanical search, and Codex can drive it.

Copy-paste prompt: bisect a regression with the failing test

User search stopped returning results sometime in the last five days. The test tests/search/search.test.ts "returns matches for a partial name" fails on HEAD.

1. Find the most recent commit from before the last five days where that test passes. Show the test output at that commit.
2. Run git bisect between that commit (good) and HEAD (bad), using `npx vitest run tests/search/search.test.ts -t "returns matches for a partial name"` as the `git bisect run` command.
3. Report the first bad commit: hash, author date, message, and the part of its diff that explains the failure.
4. Run git bisect reset, then confirm you are back on the original branch with a clean working tree.

Do not fix anything in this step.

If the test itself did not exist in older commits, have Codex copy it into a temporary file outside the repository and point git bisect run at a small script that runs it. Otherwise bisect reports “bad” for every commit that predates the test.

Pull production errors in through the Sentry MCP server

For errors that only show up in production, a stack trace pasted into the prompt loses the breadcrumbs, tags and trace around it. Sentry’s official MCP server gives Codex the issue, its events and the trace. Add it once from your terminal:

codex mcp add sentry --url https://mcp.sentry.dev/mcp
codex mcp login sentry

The server uses OAuth. For self-hosted Sentry, use the stdio package @sentry/mcp-server instead. See MCP setup for Codex for how Codex stores MCP servers. The repository getsentry/sentry-mcp had 862 GitHub stars on 2026-09-26.

Copy-paste prompt: from a Sentry issue to a failing test

Use the Sentry MCP server. Find the most frequent unresolved error in the project "payments-api" over the last 24 hours. Show me its stack trace, the release it first appeared in, and one representative event's request data with personal data redacted.

Map the top application frames to files in this repository. Then write a test that reproduces the error with inputs shaped like that event, run it, and show that it fails with the same exception type and message. Do not fix the code yet.

The last sentence keeps the loop intact. The Sentry event is evidence for step 1, not a license to patch the frame at the top of the stack.

Fix the bug in a worktree and prove it

Fix in isolation so the change arrives as its own branch. codex --worktree starts the session in a new managed Git worktree; in a session that is already running, /worktree does the same. A worktree checks out committed code only, so commit the reproduction test first.

Copy-paste prompt: minimal fix with proof

Fix the confirmed root cause: the profile update in src/routes/profile.ts has no WHERE clause on the user ID, so it updates zero rows and the handler returns the old record.

Rules:
- Make the smallest change that makes the reproduction test in tests/api/profile.test.ts pass.
- Add a guard: if the update affects zero rows, return 404 instead of 200.
- Do not change the API response shape. Do not modify, skip or loosen any existing test.
- Add one more test for the 404 path.

Then prove it, and paste the output of each step:
1. The reproduction test and the new 404 test pass.
2. The full suite passes: npm test (all of it, not only the new tests).
3. Type check and lint pass: npm run typecheck and npm run lint.
4. Revert only your change to src/routes/profile.ts, run the reproduction test, and show that it fails again. Then restore the fix.

Finish with a three-line summary: root cause, fix, evidence.

Step 4 is the one reviewers skip and the one that matters most. A regression test that also passes without the fix does not test the bug. You can check it without reading the test: the output must show red, then green.

Run the same fix headlessly

When the diagnosis is solid and you want the fix without a session, run it with codex exec. --worktree isolates the change, :workspace lets Codex write inside the worktree, and -o saves the final summary for the pull request description:

codex exec --worktree -c default_permissions=":workspace" \
  -o fix-summary.md \
  "$(cat prompts/fix-profile-persistence.md)"

prompts/fix-profile-persistence.md holds the fix prompt above. Before you open the pull request, run a local review of the change with codex review --base main. Non-interactive Codex covers codex exec output formats and CI use.

Debug intermittent failures, leaks and upgrade breakage

Some bug classes defeat a single test run. Each needs a different kind of evidence.

Intermittent failures and race conditions. One green run proves nothing. Ask for a pass count:

Copy-paste prompt: intermittent test failure

The test "processes concurrent orders" in tests/services/order.test.ts fails intermittently with "Expected 2 but received 1".

1. Run it 50 times in a loop and report the failure count as a baseline.
2. Read OrderService in src/services/order.ts and identify shared state that two concurrent calls can read and write. Explain the interleaving that produces 1 instead of 2.
3. Fix it with a real synchronization mechanism (a transaction with the right isolation level, a row lock or an atomic update). Do not add sleeps, retries or longer timeouts to the test.
4. Run the test 50 times again and report the new failure count. Anything above zero means the fix is not done.

If the failure disappears whenever Codex adds logging, the timing changed. Ask for a counter or an assertion at the suspected interleaving point instead of log lines, and keep it only for the investigation.

Memory leaks. Give Codex the growth rate and the time window, and ask for a retention path, not a list of suspects:

Copy-paste prompt: memory leak investigation

Our Node.js service's heap grows by about 50 MB per hour under steady load. It started within the last two weeks.

1. List the commits from the last two weeks that touch src/services/ or src/lib/.
2. In those diffs, look for event listeners that are never removed, timers or subscriptions that are never cleared, module-level maps or arrays with no eviction, and closures that capture request objects.
3. Pick the most likely candidate and explain the retention path from a GC root to the leaked objects.
4. Write a test that creates and tears down the component 1,000 times and asserts that the listener count, the map size or the heap usage stays bounded. Show that it fails before your fix and passes after.

Breakage after a dependency upgrade. State the version change and forbid the easy way out: “CI fails after upgrading TypeScript from 5.3 to 5.5. Read the release notes for the versions in between, fix the type errors in src/types/, and do not downgrade TypeScript or add @ts-ignore. Run the type checker after each file.” If Codex needs release notes from the web, start the session with --search, which turns on live web search.

Environment-specific bugs. When a failure only appears in a pinned runtime or with a particular dataset, run the reproduction in Codex cloud, where the environment’s setup defines the container. --attempts asks for several independent attempts (best-of-N), which helps when the first theory is often wrong:

codex cloud exec --env ENV_ID --attempts 3 \
  "The expired-subscriptions cron job skips records on Node 20. Reproduce it with tests/fixtures/expired-subscriptions.json, write a failing test, find the root cause, fix it, and report the test output before and after."
codex cloud status TASK_ID      # poll until the task finishes
codex cloud diff TASK_ID        # inspect the result
codex cloud apply TASK_ID       # apply it to your working tree

codex cloud is marked [EXPERIMENTAL] in Codex CLI 0.157.1. TASK_ID is printed by codex cloud exec.

Red CI on a pull request. Mention @codex on the pull request with the same rules: read the failing job’s log, reproduce, fix the code under test, and do not loosen tests or CI configuration. The full copy-paste comment and the repository setup it needs are in Connect Codex to GitHub.

Add a debugging skill so the method survives every session

Prompts drift between people. A skill makes the reproduce-then-diagnose discipline the default. The systematic-debugging skill from obra/superpowers runs four phases: root-cause investigation, pattern analysis, hypothesis testing, and implementation. It installs for Codex with the skills CLI (version 1.7.0 checked on 2026-09-26):

npx skills add obra/superpowers --skill systematic-debugging -a codex

It had about 272,000 skills.sh installs in the 2026-09-26 LinklyAI/best-skills snapshot of that listing (SECONDARY). Read the installed SKILL.md before you rely on it. Open issue #2277 (2026-09-10) reports that the skill directory ships test fixtures that are copied into the agent’s skill folder. For other testing and debugging skills, see testing and debugging skills.

If you prefer not to add a skill, put the loop in AGENTS.md instead: “For any bug, write a failing test and show its output before editing source files.” Codex loads AGENTS.md at the start of every session in a trusted project.

How do you know a Codex fix is right without reading every line?

The fix is a claim. These checks turn it into evidence you can confirm in minutes, whoever wrote the code.

Evidence	What it proves	Who checks it
Reproduction test output: red before the fix, green after	The test captures the reported behaviour and the fix changes it	The developer who owns the bug, from the pasted output
The reproduction test fails again when the fix is reverted	The test is not passing by accident	Same, from step 4 of the fix prompt
Full test suite, type check and lint, run in CI on the pull request	Nothing else broke	Branch protection; no human needed
Pass count over 50 or more runs, for intermittent bugs	The race is gone, not hiding	The developer, from the loop output
codex review --base main or @codex review findings	A second pass looked for side effects in the diff	The developer triages; findings with a line number get addressed
A diff limited to the files named in the diagnosis	The fix stayed in scope	The reviewer, from the file list, not the full diff

The human decision is narrow: does the reproduction test match the user’s report? If it does and every row above is green, approve on evidence. If the diff touches files the diagnosis never mentioned, send it back and ask why before you read it. For the review side of this, see review strategies for Codex changes.

When Codex debugging stalls and how to recover

Codex cannot reproduce the bug. The bug depends on state that your dev environment lacks: specific data, a third-party response, production traffic. Give Codex the exact error, the stack trace, the triggering request and the first bad release, and ask for a test that constructs that state with fixtures or a mock of the third-party call. If the state is environmental, move the reproduction to Codex cloud with an environment whose setup seeds it.

The fix treats the symptom. Codex wraps the failing line in a null check or a try/catch, and the test passes. Ask: “Why is the value null at this point? Trace it back to where it is set.” Reject fixes that add defensive code without a diagnosis that explains the bad value.

Codex changes the test instead of the code. An assertion gets loosened, a case gets .skip, or a timeout doubles. Say in every fix prompt that tests must not be modified, skipped or loosened, and check the file list: a fix whose diff touches only test files is not a fix.

The fix breaks other tests. Codex ran only the new test. Name the full-suite command explicitly (npm test, not “run the tests”) and require its output in the summary.

Codex blames the wrong commit. Reading git log invites plausible stories. Replace the story with git bisect run and the failing test, as in the bisect prompt above.

The intermittent failure disappears while debugging. Logging changed the timing. Use counters or assertions, run the test in a loop of 50 or more, and compare failure counts, not single runs.

The session loses the thread. After a long investigation, Codex repeats hypotheses it already ruled out. Ask for a summary of what is confirmed and what is ruled out, then start a fresh session with that summary as the first message. codex fork --last (run from a new terminal) forks your most recent session, so you keep its history while you try a different theory. For keeping context sharp, see context patterns for Codex.

The cloud fix passes in the container but fails locally. The cloud environment and your machine run different versions. Pin the runtime and dependency versions in the environment’s setup to match CI, and treat CI as the tie-breaker.

An @codex mention gets no response. The repository is not connected to Codex, or it has no cloud environment. Check the setup in Connect Codex to GitHub. If the problem is Codex itself (sign-in, sandbox or approval errors), see Codex error recovery.

Where to go next with Codex debugging

Test generation and execution with Codex Turn each reproduction test into lasting coverage and a suite you trust.

Codex cloud environments Build the environment that environment-specific reproductions run in.

Debugging patterns for any agent Tool-neutral recipes for stack traces, logs and flaky tests.

AI-powered incident response When the bug is a live incident: triage, mitigation and the post-incident loop.

The same workflow in the other tools: debugging with Claude Code and bug hunting in Cursor.