Level 4: You Write the Specs

Level 4 of the autonomy ladder is the stage where a developer writes the specification, a machine-checkable stop condition and the tests that decide it, and the agent then runs unattended for hours. The human judges the result by its evidence, not by reading every diff. Level 4 holds only while the agent cannot edit the checks that define done.

This page is for developers who already run agents in parallel at Level 3 and want runs they can leave, and for tech leads who decide which work may run that way. You start a large migration at six in the evening. At eight the next morning the agent reports “all tests pass” — and one of the passing tests is the one it rewrote at 3 a.m. Level 4 is the set of habits that makes that morning impossible. CTOs and executives can go straight to the measures section.

What you get from this Level 4 guide

• A four-point test that tells you whether you are at Level 4 or at Level 3 with the check removed.
• The four artefacts that exist before a long run starts, and the order to write them in.
• How to start a goal-directed run in Claude Code, Codex and Cursor, with the commands.
• A Stop hook that refuses to finish while the tests are red or the test files have changed.
• Two copy-paste prompts: draft and attack a stop condition, and produce the morning evidence report.
• The morning review: how to accept or reject a twelve-hour run without reading its whole diff.

Are you at Level 4? The four-point test

Dan Shapiro’s ladder (The Five Levels: from Spicy Autocomplete to the Dark Factory, 23 January 2026) describes this rung plainly: you “write a spec”, then “leave for 12 hours, and check to see if the tests pass”. You are at Level 4 when all four statements are true:

1. You write the spec and the finish line, not the code. Your working output is a spec and a condition.
2. Runs last longer than you watch them. An hour or more passes without a prompt from you.
3. A check you trust decides done. Tests, types and scripts decide, not the agent’s own summary.
4. The run cannot rewrite that check. Test files, CI config and the hook scripts are out of its reach.

If statement 4 is false, the run grades its own homework. That is Level 3 without the reviewer, the most common way this rung fails. The one map shows how the ladder relates to the lifecycle stages and factory stations.

What four artefacts exist before a Level 4 run starts?

None of them is code. Write them in this order, because each one is checked against the previous one.

1. The spec. The outcome, the constraints and what is out of scope, in a file the run reads (spec.md, an issue, a PRD). Spec-driven development covers keeping it authoritative, and acceptance criteria covers turning each requirement into a check.
2. The plan. The agent writes it in plan mode, and you read it. A plan is a page of text, not a 3,000-line diff: the cheapest review of the whole run. See plan mode across tools and Cursor’s PRD workflow. AI-DLC formalises this step: the AI drafts a plan and implements only after the team validates it.
3. The oracle. The tests, types and scripts that decide whether the spec holds, written or reviewed by you before the run. Tests written first are the only part of the oracle the agent did not author, which is the Level 4 case for test-driven development: not purity, provenance. Oracle strength covers whether those checks can catch a wrong answer at all.
4. The stop condition. One finish line stated as commands that must exit 0, plus a round or turn budget. The prompt below drafts it and then attacks it.

Copy-paste prompt: draft a stop condition, then attack it

Read spec.md and the plan we agreed. Draft the goal text for this run as
five blocks and nothing else.

Objective: one sentence, one finish line.
Read first: the exact files that define correct, including one
already-correct example in this repository.
Done when: the commands that must exit 0 (tests, typecheck, lint), plus one
grep proving the old pattern is gone. Say that each command's output must
appear in the transcript.
Never touch: the test files that make up the oracle, listed by path, plus
CI config, .claude/run-base, and .claude/ or .codex/ settings.
Budget: one line appended to RUN_LOG.md per round, and "or stop after 30
turns and report what is left" as the last clause.

Then attack your own draft. List every way the condition could be satisfied
by weakening a test, deleting a case, skipping a suite, or claiming a result
without running the command. Rewrite the wording so none of them works, and
print only the final version.

Stripe’s in-house agents show what a bounded condition looks like at scale: “at most two rounds of CI”, after which the branch goes back to a human (Stripe engineering blog, 9 February 2026). A budget is part of the condition, not an afterthought.

How do you start a goal-directed run in Claude Code, Codex and Cursor?

All three tools have a /goal command. They differ in who checks the condition and what else you can attach to it.

Claude Code

Checked against Claude Code 2.1.283 (latest channel) and its /goal documentation on 26 September 2026.

# In the session: plan first, approve the plan, leave plan mode (Shift+Tab to auto mode)
/plan
# then set the goal; it starts a turn at once
/goal all tests in test/billing pass, `npm run typecheck` exits 0, no file under test/ is modified, or stop after 30 turns
/goal          # status: condition, turns, token spend, last verdict
/goal clear    # stop early

After each turn, a small, fast model (Haiku by default on the Claude API) judges whether the condition holds. It returns “not yet met”, “met” or “impossible”. The condition can be up to 4,000 characters, and one goal is active per session. A goal does not change your permission mode, so a goal set while plan mode is still on cannot edit anything. Switch to auto mode first to let turns continue unattended.

The evaluator “doesn’t run commands or read files independently”. It reads the transcript. That is why the condition above names the commands, and why the Stop hook in the next section runs them itself.

For a run from a script or CI, headless mode runs the whole loop in one call. A claude -p session starts in Manual mode, not auto (v2.1.283), so pass the permission mode explicitly or the run stalls on its first unapproved tool call:

# Terminal: auto mode for unattended turns; stream each turn so a long goal does not look stuck
claude -p "/goal CHANGELOG.md has an entry for every PR merged this week" \
  --permission-mode auto --output-format stream-json --verbose

For a tighter run, replace --permission-mode auto with --allowedTools and list only the commands the goal needs.

Codex

Checked against codex-cli 0.157.1 on 26 September 2026: goals is a stable feature and on by default, and /goal sets or views “the goal for a long-running task”. Since 0.151.0, tokens spent by nested subagents count toward the goal’s budget.

# In the Codex TUI: plan first, approve the plan, then set the goal
/plan
/goal all tests in test/billing pass, `npm run typecheck` exits 0, no file under test/ is modified, or stop after 30 turns and report what is left

Codex’s own guidance on writing the objective (Long-running work page, checked 28 August 2026) asks for three parts: “Outcome, Constraints, Verification”. Codex also has a Stop hook event among its 12 hook events; hooks need persisted trust before they run. Codex has no /loop command (checked 28 August 2026 in the developer-commands list, and absent from the 0.157.1 binary on 26 September 2026).

Cursor

Cursor announced /goal in its 19 August 2026 changelog: “Use /goal to give the agent a long-lived objective to work towards until it’s fully complete.” Its CLI slash-commands reference still marked the command “Rolling out” when checked on 28 August 2026. Plan Mode “creates detailed implementation plans before writing any code”, and Cursor hooks run “before or after defined stages of the agent loop and can observe, block, or modify behavior” (all checked 28 August 2026; cursor.com was unreachable from the writing environment on 26 September 2026). Check Cursor’s hooks documentation for the current event names before you script a stop gate.

/loop (a bundled skill since 3.5, 20 May 2026) re-runs a prompt on a local schedule until an outcome is reached or you stop it; /goal works toward one finish line without a schedule.

The /goal command walks through each tool in depth, and the /loop command covers cadence-driven runs.

How do you stop a run from finishing while the tests are red?

Use a hook: a script the tool runs at a fixed point in the loop, whatever the model decides. In Claude Code, a Stop hook runs when the agent tries to end its turn. Exit code 2 blocks the stop and sends your stderr text back to the agent as the reason to continue. Unlike the /goal evaluator, the hook runs the commands itself.

Before the run starts, record the commit it starts from, so the hook can compare against it however many times the agent commits:

git rev-parse HEAD > .claude/run-base

#!/usr/bin/env bash
# .claude/hooks/require-green.sh — Stop hook: no finishing while the oracle is red
cd "${CLAUDE_PROJECT_DIR:-$(git rev-parse --show-toplevel)}" || exit 0
[ -f .claude/run-base ] || exit 0   # armed only during a recorded run

# 1. The oracle must be untouched: the run may not edit the tests it is graded by.
#    Diff against the commit recorded before the run, not HEAD, so a committed
#    test edit still counts, and list new untracked test files too.
base=$(cat .claude/run-base)
changed=$( { git diff --name-only "$base" -- test/; git ls-files --others --exclude-standard -- test/; } )
if [ -n "$changed" ]; then
  printf 'Revert your edits to the test files and fix the code instead:\n%s\n' "$changed" >&2
  exit 2
fi

# 2. The oracle must be green.
if ! out=$(npm test --silent 2>&1); then
  printf 'Tests are failing. Fix the code, not the tests:\n%s\n' "$(printf '%s' "$out" | tail -n 40)" >&2
  exit 2
fi
exit 0

Register it in .claude/settings.json (make the script executable with chmod +x):

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          { "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/require-green.sh", "timeout": 600 }
        ]
      }
    ]
  }
}

The hook arms itself only during a recorded run: without .claude/run-base it exits 0 silently, so ordinary sessions in the same repository see no hook errors. Delete the file when the run is reviewed. Three limits to know before you rely on it. Claude Code applies an eight-consecutive-continuation cap: after stop hooks have continued the turn eight times in a row, the next block is overridden and the turn ends (CLAUDE_CODE_STOP_HOOK_BLOCK_CAP raises it). The settings file lives in the repository, so the run can edit it too; list it under “Never touch” and check it in CI. And /goal is itself a Stop hook, so it is unavailable when disableAllHooks is true. The command tells you so instead of failing silently.

The same gate in the other tools:

• Codex runs the same script from its Stop hook, but do not copy the Claude Code registration. Codex 0.157.1 does not set CLAUDE_PROJECT_DIR, so ${CLAUDE_PROJECT_DIR}/.claude/hooks/require-green.sh expands to /.claude/hooks/require-green.sh, the shell exits 127 with “not found”, and the gate never runs. Codex runs the command through a shell in the session’s working directory, so resolve the repository root with git in .codex/hooks.json:

  {
    "hooks": {
      "Stop": [
        {
          "hooks": [
            { "type": "command", "command": "\"$(git rev-parse --show-toplevel)/.claude/hooks/require-green.sh\"", "timeout": 600 }
          ]
        }
      ]
    }
  }

  Then open /hooks and trust it: project hooks do not run until trust is persisted, and again after anyone changes the file (Codex 0.157.1). Agent hooks across tools covers the rest of the Codex registration.

• Cursor hooks can block the agent loop, but its event names were not verifiable from the writing environment on 26 September 2026. Check them in Cursor’s hooks documentation before you port the script.

Hooks in Claude Code and agent hooks across tools cover the other events. Protecting the oracle goes further: locked test paths, CODEOWNERS, and holdout checks kept out of the repository.

How do you check a twelve-hour run without reading every line?

You read evidence, in a fixed order, and read code only where evidence is not enough. The agent assembles the evidence; you decide.

1. Re-run the oracle yourself, or read the CI result on the pushed branch. Never accept the run’s own report of green.
2. Check that the oracle is intact. git diff --stat $(cat .claude/run-base) -- test/ .github/ .claude/ must be empty unless the spec asked for test changes.
3. Read the spec delta. Which acceptance criteria are met, and which ones the agent says it could not meet.
4. Read RUN_LOG.md. Look for the round where progress stalled or the approach changed. Long runs degrade: SlopCodeBench (arXiv, 25 March 2026) had 15 coding agents extend their own code over 196 checkpoints, found that “no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints”, and measured the decay as structural erosion and verbosity.
5. Read code only for the escalation classes. Schema and migrations, auth, money, and anything a customer sees. Reading evidence instead of code defines these classes.
6. Sign off or return. The developer who wrote the spec signs off on the evidence. A change in an escalation class also needs its code owner.

Copy-paste prompt: the morning evidence report

The run toward the goal in RUN_LOG.md has ended. Do not change any code.
Produce EVIDENCE.md with these sections and nothing else:

1. Oracle: run the full test suite, the typecheck and the lint step now,
   and paste each command with its exit code and last 10 lines of output.
2. Oracle integrity: the output of
   git diff --stat $(cat .claude/run-base) -- test/ .github/ .claude/
   and one sentence on every file it lists.
3. Spec coverage: a table of every acceptance criterion in spec.md with
   met / not met / partly met and the test that proves it.
4. Escalation classes: every changed file that touches migrations, auth,
   payments or user-visible copy, with one line on what changed.
5. Turning points: the rounds in RUN_LOG.md where the approach changed or
   progress stalled, and why.
6. Doubts: what you are least sure of, and the check that would settle it.

The same EVIDENCE.md, attached to a pull request, is the evidence bundle that reviewers read instead of the diff.

What should tech leads, CTOs and executives measure at Level 4?

Four measures, per repository or per change class, show whether Level 4 is working or whether runs only look finished. Adopt the definitions as written and set thresholds from your own baseline.

Measure	Definition	Warning signal
Run acceptance rate	Unattended runs whose output merged without rework, divided by all runs started	Falls as runs get longer
Oracle-edit rate	Runs whose diff touched test, CI or hook files without the spec asking for it	Any rise
Escaped defects per class	Production incidents traced to changes approved on evidence, per change class	A class that keeps escaping goes back to Level 3
Budget overrun	Runs stopped by the turn or token budget rather than by the condition	Conditions that are unreachable or vague

The oracle-edit rate is the one to watch. A run that edits its own checks has not reached Level 4, whatever the acceptance rate says.

What breaks when a twelve-hour run comes back green?

The agent edited the oracle

Symptom: the suite is green and a test file is in the diff. Recovery: revert the test edits, re-run the oracle on the result, and add the paths to the hook and to “Never touch”. Keep the decisive check in CI, where the run cannot rewrite it.

The evaluator believed a claim

Symptom: the goal is “met”, but nobody ran the tests in the final turn. Recovery: write “the output of npm test must appear in the transcript” into the condition, and add the Stop hook so the check runs whatever the agent says.

The run wandered or thrashed

Symptom: the log shows the same fix tried three ways, or work outside the spec. Recovery: split the objective in two, name one already-correct example to copy, and restart from the last good commit in the log, not from the end state.

The budget ran out before the goal did

Symptom: the quota is gone and the condition is still unmet. Recovery: the condition was unreachable or too broad. Add a turn clause (“or stop after 30 turns and report”), and prove the condition can pass by running it on a hand-made fix first.

The run never started

Symptom: /goal refuses to start in Claude Code. Recovery: read the message. The usual cause is hooks being disabled (disableAllHooks), managed-hooks-only policy, or an untrusted folder, since /goal runs through the hooks system.

Green, but wrong

Symptom: every check passes and the feature still misbehaves. Recovery: the oracle is too weak for this class. Strengthen it with a failing test for the bug, and move the class back to reading code until the new checks have caught something on their own.

When is a change class ready for Level 5?

Level 5 removes the human from the loop for whole stations, not single runs. A change class is ready when unattended runs in that class keep a high acceptance rate, the oracle-edit rate is zero, escaped defects stay flat, and the specs, not the prompts, are what your team reviews. Everything a stop condition cannot decide (intent, taste, the escalation classes) stays with people; the human’s job lists it.

Compound engineering keeps each run from starting at zero: every finished run leaves a rule, a check or a skill behind. See compound engineering, the packaged discipline in Superpowers, and grill me as the alignment pass to run before a long job.

Where to go next from Level 4

Level 5: You Run the Software Factory Six stations, and what each tool documents for them.

Reading evidence instead of code What a human reads once the diff stops being the unit of review.

Protecting the oracle Checks the agent cannot edit: locked paths, CI-owned gates, holdouts.

The /goal command Goal-directed autonomous runs, tool by tool.

Spec-driven development Keeping the spec, not the code, as the thing you review.

Find your level Twenty-five questions that place your workflow on the ladder.