# AI Developer Toolkit — Autonomy Ladder (free, no account required) > The "Autonomy Ladder (free, no account required)" section of the AI Developer Toolkit documentation (English). Full site index: https://developertoolkit.ai/llms.txt Generated: 2026-11-02T15:41:09.101Z Source: https://developertoolkit.ai Total documents: 11 --- # The Autonomy Ladder: Which Level Is Your Workflow At? URL: https://developertoolkit.ai/en/ladder/ Description: Dan Shapiro's six-level scale for how much code agents write and humans read, a test to place each loop on it, and the guide for every rung. The autonomy ladder is Dan Shapiro's six-level scale, Level 0 to Level 5, for how much of a team's code coding agents write and how much of it humans still read. Each level describes one loop. From Level 3 upward, a loop climbs only when a machine-checkable signal, such as commands that exit 0, replaces a human reading the diff. Your team uses Claude Code, Codex or Cursor daily, yet every change still waits for someone to read the diff. Developers, tech leads, CTOs and executives each ask "where are we?" and mean something different. This page gives all four one scale, a five-question test, and the guide for each rung. ## What does each level of the ladder look like? | Level | Site name (Shapiro's name) | Who writes | Who reads | In Shapiro's words | |-------|---------------------------|------------|-----------|--------------------| | L0 | By hand (Spicy autocomplete) | Human | Human, every character | "not a character hits the disk without your approval" | | L1 | Assisted (The coding intern) | Human, mostly | Human, all of it | "you offload specific, discrete tasks to your AI intern" | | L2 | Paired (The junior developer) | Human and agent, paired | Human, every line | "feels like you are done. But you are not done" | | L3 | Review manager (The developer) | Agent, most of it | Human, as full-time reviewer | "Your life is diffs." | | L4 | Spec manager (The engineering team) | Agent, from a spec | Tests, mostly | "leave for 12 hours, and check to see if the tests pass" | | L5 | Dark factory (The dark software factory) | Agent | Nobody | "It's a black box that turns specs into software." | *Source: Dan Shapiro, "The Five Levels: from Spicy Autocomplete to the Dark Factory", 23 January 2026; level names as quoted by Simon Willison, 28 January 2026. The short names are this site's.* Shapiro puts 90% of "AI-native" developers at Level 2, says "almost everyone tops out" at Level 3, and places "a handful of people" at Level 5. Levels 0 and 1 sit at the light end of [AI-assisted coding](/en/glossary/ai-assisted-coding/), and this site places [AI pair programming](/en/glossary/ai-pair-programming/), the phrase GitHub Copilot popularised, at Level 2. See [the evidence behind the ladder, sourced and dated](/en/ladder/state-of-agentic-engineering-2026/). ## Which rung is each of your loops on? A **loop** is a repeatable class of change with its own trigger, check and stop condition: dependency bumps, flaky-test repair, feature work in one service. Answer per loop, against the repository you ship from. The first question you cannot answer yes *with evidence* is that loop's ceiling. 1. **L0 to L1.** Did code reach disk this week that you did not type? 2. **L1 to L2.** Do you hand the agent whole tasks with acceptance criteria, not completions inside a function you are writing? 3. **L2 to L3.** Does the agent run long enough, and often enough, that you first meet its work as a diff? 4. **L3 to L4.** Is there a stop condition a machine can evaluate, such as commands that exit 0, so you can leave the run and check the result? 5. **L4 to L5.** Does anything merge that no human read, and can you name the oracle that made that safe? Evidence is an artifact: a CI job, a test command, a merged pull request. The [one map](/en/ladder/one-map/) lists the evidence each level requires at each lifecycle stage and defines a team's level as the distribution of its merged changes across loop levels. **Copy-paste prompt: place this repository's loops on the ladder** ```text Read the last 30 days of git history, the CI configuration and the test commands in this repository. Group the merged changes into loops (repeatable classes of change, for example dependency bumps, bug fixes, feature work per service). For each loop, answer five questions with evidence: 1) does agent-written code reach disk, 2) does the agent receive whole tasks with acceptance criteria, 3) is its work first seen as a diff, 4) is there a stop condition a machine evaluates (name the exact command that must exit 0), 5) does anything merge unread and which check made that safe. The first question without evidence is the loop's level. Output a table: loop, level, the evidence (file path or command), and the one missing artifact that blocks the next level. Do not guess: write "no evidence" where you found none. Do not change any files. ``` ## How do you move a loop from Level 3 to Level 4? Level 3 is where review becomes the bottleneck; Level 4 is defined by leaving. The move needs a stop condition that runs without you: commands the agent must make pass before it stops. **Claude Code** Use `/goal` interactively, or run the loop headless in CI with `claude -p "Bump dependencies until npm test exits 0" --output-format json --permission-mode dontAsk --allowedTools "Edit Bash(npm install *) Bash(npm update *) Bash(npm test)" --disallowedTools "Edit(**/tests/**) Edit(**/*.test.*)"` so the job can parse the result. Headless runs start in Manual permission mode (auto mode on third-party providers or with telemetry off, from v2.1.285, where a classifier decides) and, with `--permission-mode dontAsk`, refuse any tool call that would need approval (edits, shell commands) unless it is on the allow list; read-only tools such as Read and Grep still run. So grant exactly what the loop needs: the install commands that change dependencies and the test command that checks them. Deny rules are evaluated before allow rules, so the agent can edit source but not the test files that are its stop condition (Claude Code v2.1.283). See [goal-directed runs with `/goal`](/en/workflows/automate/goal-command/). **Codex** Use `/goal` in the terminal UI, or `codex -a never exec -s workspace-write -c sandbox_workspace_write.network_access=true --output-schema SCHEMA_FILE "Bump dependencies until npm test exits 0"` in CI. SCHEMA_FILE is the path to your JSON Schema. Approval flags go before the subcommand. `-s workspace-write` lets the run edit the checkout and run the stop-condition commands, but blocks network access by default, so without the `-c` override `npm install` cannot reach the registry. `--output-schema` makes the final answer follow a JSON Schema your job checks (Codex CLI 0.157.1). `workspace-write` makes the whole checkout writable, tests included, so the job-side check below is what catches an edited test. **Cursor** Cursor's changelog lists `/goal` as "Rolling out" from 19 August 2026 (checked 2026-08-28). Where `/goal` has not reached your account yet, run the loop as a Cloud Agent and require a CI check to pass before anyone opens the pull request. **Copy-paste prompt: write the stop condition for one loop** ```text Take the dependency-bump loop in this repository. Write its stop condition as a list of shell commands that must all exit 0 before the agent may stop: install from the lockfile, type check, lint, the full unit test suite, and a build. Run each command now and report which pass on the current main branch. For any command that fails on main, stop and tell me: the stop condition cannot be stronger than the baseline. Then list what these commands do not check (runtime behaviour, security advisories, breaking API changes) so I can decide what else a human must read. ``` A stop condition the agent can edit, such as a test it may rewrite, is not a stop condition. Path deny rules do not close every route: the agent still edits `package.json`, including the script `npm test` runs, and `npm install` runs the install scripts of every new dependency. So after the agent exits, the CI job re-runs the stop-condition commands itself, fails if the diff touches a test file or the `test` script, and runs on a disposable runner with no deploy secrets in its environment. [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) covers what a human reads instead of the diff, and which changes (auth, money, schema migrations) still get read line by line. The tech lead who owns the loop signs off its promotion, and records the stop-condition commands and the last month of rejected diffs they would have caught in the loop register. ## Where does placing a loop on the ladder go wrong? - **One level for the whole team.** Averaging hides the loops that are ready to move. Recovery: keep a register with one row per loop, as in the [one map](/en/ladder/one-map/). - **Claiming a level from confidence.** A loop "at Level 4" whose stop condition is "looks fine to me" is at Level 3. Recovery: name the command; if you cannot, demote the loop. - **Skipping Level 3.** Merging unread before you know how the agent fails produces incidents. Recovery: stay at review until your checks would have caught last month's rejected diffs. - **Mixing in other maturity models.** JetBrains' AIDEs L1–L5 and Every's five stages define levels differently. Recovery: do not map them onto the ladder. ## Where to go next on the autonomy ladder Follow your role track ([developer](/en/start/developer/), [tech lead](/en/start/tech-lead/), [CTO](/en/start/cto/), [executive](/en/start/executive/)) or go to your loop's rung. - [One map: ladder, lifecycle and stations](/en/ladder/one-map/) — The evidence each level requires at each stage, and how to report a team's level. - [Level 1–2: Assisted and Paired Coding](/en/ladder/level-1-2-assisted-paired/) — Per-tool setup that gets a repository from completions to delegated tasks. - [Level 3: You Review the Diffs](/en/ladder/level-3-review-diffs/) — Parallel agents, review automation and the review ceiling. - [Level 4: You Write the Specs](/en/ladder/level-4-write-specs/) — Plan mode, goal-directed runs, hooks and stop conditions that make leaving safe. - [Level 5: You Run the Software Factory](/en/ladder/level-5-run-the-factory/) — Six factory stations mapped to what Claude Code, Codex and Cursor document. - [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) — What a human reads instead of the diff, and when someone still reads code. - [The human's job in agentic engineering](/en/ladder/the-humans-job/) — Taste, architecture, direction, stop conditions, permissions and what not to automate. - [Your craft and career when agents write the code](/en/ladder/craft-and-career/) — Which skills compound, which erode, and a plan for each rung. --- # Your craft and career when agents write the code URL: https://developertoolkit.ai/en/ladder/craft-and-career/ Description: Which engineering skills compound and which erode when agents write the code, the paradox of supervision, and a portfolio plan for each ladder level. When coding agents write most of the code, the compounding skills are specification, verification design, architecture, and taste. Syntax recall and boilerplate can be let go; unassisted debugging erodes most visibly (the largest gap in Anthropic's January 2026 trial). Anthropic's study of its own engineers names the trap the "paradox of supervision": supervising an agent takes the same coding skills that heavy delegation wears away. You merged 30 pull requests last month and typed almost none of them. Your performance review still asks what you built, and an incident just landed in a module no human has read in full since March. This page is for developers who want an honest answer to "what is my craft now", and for tech leads who have to answer it for other people. ## What this page gives you for your craft and career - The skills that compound and the skills that erode, with a source for each claim. - The "paradox of supervision", and the habits the one controlled study links to keeping what you learn. - Setup for a learning mode in Claude Code, Codex, and Cursor. - A portfolio plan for each level of the [autonomy ladder](/en/ladder/): what to keep sharp, what to hand over, and what evidence to collect. - A one-page skills plan you can copy, and four prompts that turn your own git history into that evidence. - Self-checks that show whether your skills are holding, and a list of what no source can tell you yet. ## What the evidence says about skills when agents write the code Two studies speak directly to the question. Both come from Anthropic, a model vendor, so read them as a vendor's research about its own product and staff. No independent study of comparable design was available when this page was written. **The internal workplace study.** In August 2025 Anthropic surveyed 132 of its engineers and researchers and ran 53 in-depth interviews. The results were published on 2 December 2025 by Saffron Huang and colleagues. Most respondents said they could "fully delegate" only 0–20% of their work. One engineer described their work as having shifted "70%+ to being a code reviewer/reviser rather than a net-new code writer." Some worried about "skills atrophying as [they] delegate more" and about losing the "collateral" learning that happens during manual problem-solving. The report names the risk directly: > "effectively using Claude requires supervision, and supervising Claude requires the very coding skills that may atrophy from AI overuse." A senior engineer in the same study explained why seniority protects them: "I'm primarily using AI in cases where I know what the answer should be or should look like. I developed that ability by doing SWE 'the hard way'." A junior engineer, in their view, would need "a lot of deliberate effort to continue growing my own abilities rather than blindly accepting the model output." **The skill-formation trial.** On 29 January 2026 Judy Hanwen Shen and Alex Tamkin published a randomized study of 52 mostly junior engineers. Each engineer learned Trio, a Python library for asynchronous programming, by building two features. The group that used an AI assistant averaged 50% on the follow-up quiz. The group that coded by hand averaged 67%. The AI group finished about two minutes faster, but that difference was not statistically significant. The biggest gap was on debugging questions. The authors note the limit themselves: the quiz measured comprehension shortly after the task, and the study "does not resolve" whether that predicts long-term skill. Participants who handed over all the code writing or the debugging scored lowest. Those who asked conceptual questions, or asked follow-up questions about generated code, scored highest. The finding is not "AI makes you worse". It is that **how you delegate decides what you keep**. Simon Willison proposed [vibe engineering](/en/glossary/vibe-engineering/) for experienced engineers who go faster with agents while staying accountable for what ships; he later said agentic engineering was winning as the name. ## Which skills compound when agents write the code? These four skills get more valuable as agents write more of the code. Each one either decides what the agent builds or decides whether its output can be trusted. [The human's job](/en/ladder/the-humans-job/) covers them as responsibilities; this section covers them as skills you practise. | Skill | What it looks like in practice | Why it compounds (source) | How to practise it this month | | --- | --- | --- | --- | | **Specification** | Turning intent into acceptance criteria and a stop condition an agent cannot misread | Karpathy: "People have to be in charge of the spec and plan" (his published transcript of his Sequoia Ascent talk, 30 April 2026). Shapiro's Level 4 developer "write[s] a spec" and checks whether the tests pass (January 2026) | Write the [acceptance criteria](/en/workflows/plan/acceptance-criteria/) before every agent task, then count how often the result matched them on the first run | | **Verification design** | Choosing the oracle that decides "done" and keeping it out of the agent's reach | Sonar, January 2026: "96% of developers do not fully trust AI-generated code, and only 48% always verify it before committing." That gap is the skill | For each feature, write the one check the agent cannot edit, and read [how strong your oracle is](/en/workflows/test/oracle-strength/) | | **Architecture** | Boundaries, dependency direction, and what may change without a human decision | DORA 2025: "Teams working in loosely coupled architectures with fast feedback loops see gains, while those constrained by tightly coupled systems and slow processes see little or no benefit" | Record one [architecture decision](/en/workflows/plan/architecture-decisions/) a week, as a file the agent reads | | **Taste** | Seeing that a working implementation is the wrong one | Karpathy, same transcript: "You still have to be in charge of aesthetics, judgment, taste, and oversight." SlopCodeBench (March 2026) measured "structural erosion" as agents extend their own solutions, decay that a passing test suite does not report | Weekly, reject one green agent result on design grounds and write down why | A fifth skill sits under all four: **a working model of the system in your head.** An engineer in Anthropic's study described building it by reading "docs and code that isn't directly useful for solving your problem". It is the first thing heavy delegation removes. ## Which skills erode, and which of them can you let go? Not everything that erodes is a loss. The honest split is between skills the agent now does better and more cheaply, and skills that only look like typing. | Skill | Verdict | Reason | | --- | --- | --- | | Syntax and standard-library recall | Let it go | The agent has it and gates catch the errors. | | Boilerplate, scaffolding, and glue code | Let it go | Agents handle it best. Keep an eye out for duplicated code. | | Exploring a new tool's configuration by hand | Let most of it go | One engineer in Anthropic's study: "now I rely on AI to tell me how to use new tools and so I lack the expertise." Keep hands-on depth for the two or three tools you are accountable for. | | Debugging from first principles | **Protect it** | The largest gap in the skill-formation trial, and what incidents need once the agent's fix has failed twice. | | The "collateral" learning from struggling with a problem | **Protect it deliberately** | Anthropic's study names its loss as the core worry. No gate replaces it. | | Reading every diff line by line | Change its shape | It does not scale: Faros AI (April 2026, 22,000 developers) measured 31.3% more pull requests merging with no review at all. Replace it with [reading evidence instead of code](/en/ladder/evidence-not-diffs/), keeping code reading for high-risk changes. | ## The paradox of supervision, and how you stay out of it The paradox works like a ratchet: delegating erodes the skill you need to check what you delegated, weaker checking means more trust, and more trust means more delegation. The way out is not to delegate less. It is to change **how** you delegate, and to schedule the practice that delegation removes. The skill-formation trial and Anthropic's workplace study point to four habits: 1. **Ask for understanding, not only for output.** Participants who asked conceptual questions or asked follow-up questions about generated code kept the most. Turn that into a rule for the session (the setup below does this). 2. **Predict before you read.** Before you open the agent's diff, write two lines saying what you expect it to change. A wrong prediction points at a gap in your model of the system, and that gap is worth more than the diff. 3. **Solve some problems by hand on purpose.** One engineer in Anthropic's study: "Every once in a while, even if I know that Claude can nail a problem, I will not ask it to. It helps me keep myself sharp." Put a fixed slot on your calendar for this, not "when there is time". 4. **Debug first, delegate second.** When a failure lands, spend 15 minutes forming your own hypothesis before you hand it to the agent. Then compare your hypothesis with what the agent found. ### Set up a learning mode in your tool Only Claude Code ships a built-in style for this; in the other two tools you give the agent the same contract as an instruction. **Claude Code** Claude Code has built-in Explanatory and Learning output styles. The skill-formation study names "Claude Code Learning and Explanatory mode" as tools "designed to foster understanding". In a session, run `/output-style` (added in Claude Code 2.1.269) and pick **Learning** for a codebase or library you are learning, or **Explanatory** when you want reasons without extra exercises. Switch back for familiar work: the explanations cost tokens and time. **Codex** We found no built-in learning or explanatory mode in `codex --help` or `codex features list` (Codex CLI 0.157.1, checked 26 September 2026). Paste the learning contract below at the start of the session. To make it standing for a shared repository, agree with your tech lead before adding it to `AGENTS.md`, which Codex reads before any work, because it changes every teammate's sessions. **Cursor** Paste the learning contract below at the start of an Agent chat. To make it stick for a project, save it as a Rule, Cursor's mechanism for giving the agent standing instructions. Because Rules can be shared across a team, keep a personal learning rule out of the team's shared set unless the team agrees. **Copy-paste prompt: a learning contract for this session** ```text For this session I am learning this part of the codebase, not just shipping. Work under these rules: 1. Before you write code, explain your plan in five lines or fewer and name the concept in this codebase or library that the plan depends on. 2. After each change, ask me one question that checks I understood why the change works. Wait for my answer before you continue, and correct me if I am wrong. 3. If a test fails, do not fix it yet. Tell me which test failed and ask me for my hypothesis first. Then compare my hypothesis with yours. 4. Leave the final 10-20 line function of the task for me to write. Describe what it must do and which tests will check it. 5. At the end, list the three things in this change I should be able to explain to a colleague without notes. ``` **Copy-paste prompt: explain-back check after an agent task** ```text I am going to explain the change you just made, without looking at the diff. After I explain it, compare my explanation with what the code actually does. List what I got right, what I missed, and anything I got wrong, citing file and line for each point. Then give me one question about a failure mode of this change that I did not mention. Do not praise the explanation; only report the differences. ``` ## A portfolio plan for each ladder level When agents write the code, portfolio evidence moves to the artifacts that decided what got built and proved it worked. Each ladder level adds one kind of artifact. Keep the earlier ones, because a portfolio that shows only the top level looks untested. | Ladder level | Keep sharp by hand | Hand over to the agent | Portfolio evidence to collect | Signal that you are ready for the next level | | --- | --- | --- | --- | --- | | [Level 1–2: assisted and paired](/en/ladder/level-1-2-assisted-paired/) | Debugging, reading unfamiliar code, the core of your main language | Boilerplate, test scaffolding, lookups | Explain-back notes on one agent change a week; one problem a week solved by hand | You can predict most of an agent diff before you read it | | [Level 3: you review the diffs](/en/ladder/level-3-review-diffs/) | Code review judgment, spotting design drift | Implementation of well-scoped tasks | A review log: defects you caught, the class of each, and the check that would have caught it automatically | Most defects in your log have a check that now catches them | | [Level 4: you write the specs](/en/ladder/level-4-write-specs/) | Specification, oracle design, architecture decisions | Multi-hour runs against a stop condition | Pairs of spec and evidence: the spec, the acceptance criteria, the test that failed first, and the merged result; architecture decision records | Your specs come back right on the first run more often than not, and the misses teach you something new | | [Level 5: you run the factory](/en/ladder/level-5-run-the-factory/) | Taste, the decision on what may run unattended, incident debugging | Whole loops, including their review station | Harness contributions: gates, fitness functions, and the written case for each loop that runs unattended; incidents where a gate held | Other people's loops run on your gates. Level 5 is not a default career goal: Shapiro places only "a handful of people" there, in "small teams, less than five people", and a strong Level 4 portfolio is a complete one | ### Copy the one-page skills plan Save this as a private file, fill it in, and bring it to your next one-to-one. It is the exit artifact of the developer track: a written plan for the skills you keep sharp and the ones you hand over. ```markdown # Skills plan: , ## Current ladder level, per loop - Feature work in
: Level - Bug fixes: Level - Refactors and migrations: Level ## Compounding skills: one goal each this quarter - Specification: - Verification design: - Architecture: - Taste: ## Protected practice (calendar slots, not intentions) - Unassisted problem: - Debug-first rule: 15 minutes of my own hypothesis before I delegate a failure - Learning mode on for: ## Handed over on purpose - : handed over because ## Evidence collected this quarter - Spec and evidence pairs: - Review or trust log: - Decision records: - Gates or checks I added: ``` ### Prompts that turn your history into evidence **Copy-paste prompt: build a review log from recent pull requests** ```text Read the pull requests I authored or reviewed in this repository over the last 90 days (use git log and the review comments you can access). Produce a table with one row per defect I caught in an agent-written change: PR, file, the defect in one line, its class (logic, security, performance, design, test gap), and the automated check that would have caught it. Add a final column saying whether that check exists in the repository today. Do not invent defects: if a comment is ambiguous, leave it out and list it separately. ``` **Copy-paste prompt: find where your model of the system is thin** ```text Look at the modules I changed most over the last 90 days (git log --author). For each of the top five, ask me three questions a senior engineer on this codebase should answer without notes: one on data flow, one on a failure mode, and one on why the current design was chosen. Ask one question at a time and wait for my answer. At the end, rank the five modules by how well I answered, and for the weakest one propose a 60-minute exercise I do by hand. ``` ## How do you prove your skills are holding? Your judgment of your own skill is the weakest check available, so measure it. In a study of 16 experienced open-source developers on 246 issues with early-2025 tools (METR, July 2025), tasks took 19% longer with AI, yet afterwards they "still believed AI had sped them up by 20%". Use checks that produce a number or an artifact: - **Unassisted kata, monthly.** Solve one timed problem in your main language with no agent. Track the time and the number of mistakes. A steady drift upward is the early warning. - **Explain-back score.** After the explain-back prompt, count the points you missed. If the misses keep growing on familiar modules, your model of the system is thinning. - **Prediction hit rate.** Compare your two-line prediction with the actual diff. Record hit or miss. - **First-run spec match.** For Level 4 work, record whether the agent's first result met your acceptance criteria. This measures your specification skill, not the model. - **Catches that became checks.** Count the entries in your review log that now have an automated check. That count is verification design, measured. The code does not rely on your self-assessment either: it goes through tests, types, lint, [an evidence bundle](/en/workflows/ship/evidence-bundle/) on each pull request, and a human read for high-risk changes. You own the practice; your tech lead reviews the plan and evidence with you each quarter and signs off on the level you work at in each loop. ## What tech leads should change on the team The Anthropic study quotes a senior engineer: "It's been sad that more junior people don't come to me with questions as often, though they definitely get their questions answered more effectively and learn faster." Those questions used to pass on judgment. Three changes restore the channel without slowing the team: - **Pair juniors on specs, not on code.** Review a junior's acceptance criteria before the run, not only the diff after it. The spec shows what they understood. - **Make the portfolio the unit of review.** Ask for spec and evidence pairs, review logs, and decision records in one-to-ones and performance reviews, instead of pull request or line counts. [Career ladders when output is cheap](/en/org/career-ladders/) covers the matrix. - **Protect practice time in the plan.** A learning-mode slot and an unassisted problem each week cost less than one incident that nobody on the team can debug. [Junior developers on an agentic team](/en/teams/junior-developers/) and [upskilling](/en/teams/upskilling/) turn this into a curriculum. ## What goes wrong with a craft plan when agents write the code **You stop being able to debug your own system.** The symptom is that every failure goes straight to the agent, and you cannot say what the fix changed. To recover, apply the debug-first rule for a month, and run the "find where your model is thin" prompt on the module behind the last incident. **Your portfolio shows only throughput.** Merged pull request counts say nothing about judgment, and reviewers now discount them. To recover, rebuild the quarter's evidence with the review-log prompt, and attach a spec and evidence pair to each achievement you claim. **The learning mode becomes permanent.** Explanations on every task slow shipping work, and you start skimming them. Turn it on per subsystem or library, and off when your explain-back score stays high for a month. **You protect the wrong skills.** Hours drilling syntax are hours not spent on specification. If an agent and a gate cover a skill, hand it over and record the gate. **Reassurance replaces evidence.** "Engineers will always be needed" and "juniors are finished" are both unsourced. Treat any career forecast without a publisher and a date as noise. ## What no source can tell you yet No study available on 26 September 2026 measures what happens to an engineer's skills over years of agent-heavy work. The skill-formation trial measured comprehension right after the task. Anthropic's workplace study is self-report from one company. We could not retrieve and verify the employment data for early-career developers, so this page prints no number about it. What the evidence does support is narrower and more useful: how you delegate changes what you keep, and the skills that decide what gets built and whether it can be trusted are the ones agents do not take over. ## Where to go next with your craft - [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) — The review skill that replaces line-by-line reading at volume. - [The human's job in agentic engineering](/en/ladder/the-humans-job/) — The six responsibilities behind the compounding skills. - [Level 4: you write the specs](/en/ladder/level-4-write-specs/) — Where specification and stop conditions become the daily work. - [Developer track](/en/start/developer/) — The full reading path this page closes. - [Career ladders and performance reviews](/en/org/career-ladders/) — For leads: rewriting the competency matrix around specs and verification. --- # Reading evidence instead of code URL: https://developertoolkit.ai/en/ladder/evidence-not-diffs/ Description: What to read instead of an agent's diff: spec delta, acceptance results, oracle strength and runtime signals, plus the changes where a human still reads code. Reading evidence instead of code means approving an agent's change on four artifacts rather than on its diff: the spec delta, the acceptance results, oracle strength, and runtime signals from a running build. A named human still reads code for authentication, money, schema and migrations, changes to the tests themselves, and anything no check covers. The agents opened 14 pull requests overnight. Each one is green, each one is 300 lines, and you have a planning meeting at ten. Reading all of them properly takes the whole day. Skimming them takes an hour and proves nothing, because a skimmed diff is an approval with a worse audit trail. This page gives that hour a better use. ## What evidence-based review gives you - **A reading order** for any agent pull request: spec delta, acceptance results, oracle strength, runtime signals, with the question each one answers. - **An escalation table** of the change classes where someone still reads the code, with the reason each class is on it and file globs to start from. - **Five copy-paste prompts** that make the agent produce the evidence, audit its own oracle, classify the change, say what its checks would miss, and explain the code you are about to change. - **A personal trust log** that moves one change class at a time from "I read everything" to "I read the evidence". - **A way to keep shared context** of a codebase nobody reads diff by diff, as the Claude Code team describes it. - **The warning signs** that you stopped reading too early, and how to recover from each one. - For tech leads, CTOs and executives: **three metric definitions** that separate an evidence review from no review at all. ## Why reading every diff stops working The load is measured. Faros AI's *Acceleration Whiplash* report (April 2026; two years of telemetry across 22,000 developers and 4,000 teams) found median time in review up 441.5% and 31.3% more pull requests merging with no review at all. In the same data, epics completed per developer rose 66.2% while incidents per pull request rose 242.7%. Generation scaled; reading did not. Developers already distrust what they are not checking. In Sonar's developer survey (January 2026, more than 1,100 respondents), "96% of developers do not fully trust AI-generated code, and only 48% always verify it before committing." So the choice is not between reading every line and reading nothing. [Level 3 of the ladder](/en/ladder/level-3-review-diffs/) ends when review bandwidth becomes the ceiling. The way past it is to change **what** you read: output from checks that ran, instead of text that looks plausible. ## Is "did a human read the code?" the right question? Not quite. Anthropic's Claude Code team calls it "an imprecise proxy for 'what could go wrong with this change, and which check—automated or human—is the right one to catch it?'" (Anthropic engineering blog, "How the Claude Code team does code review", 15 October 2026). Ask the second question of every change: name what could go wrong, name the check that would catch it, and put a person on the change only where no check can. The escalation table below answers it once per change class. The same article reports figures from its own codebase. It says Claude writes "about 90 percent of Claude Code's own code" (the article does not say what is counted), and that from June to September 2026 merged pull requests to Claude Code's codebase "more than doubled while the share that had to be reverted fell by more than 40 percent." These are Anthropic's self-reported numbers, not an industry benchmark, and the article gives relative changes, not absolute counts. It does not tie the fall in reverts to any one change. Alongside the figures it describes the measure the team optimizes: the minimum human review time that doesn't raise reverts or incidents. The approval tiers behind it are on [governance and autonomy](/en/org/governance-autonomy/). ## What do you read instead of the diff? Read four artifacts, in this order. Each one answers a question the diff answers poorly, and each one can fail in a way you can see. | # | Artifact | The question it answers | Where it comes from | What a failing one looks like | | - | -------- | ----------------------- | ------------------- | ----------------------------- | | 1 | **Spec delta** | Did the agent change the behaviour we asked for, and only that? | The spec or ticket, plus a short, plain-language list of behaviour changes | Behaviour changes nobody asked for; "refactored X while I was there" | | 2 | **Acceptance results** | Does each acceptance criterion have a check that ran and passed? | Test, e2e and eval output, mapped one criterion to one check | A criterion with no check, or a check that was skipped | | 3 | **Oracle strength** | Could those checks have failed, and did the agent touch them? | The test diff, mutation results on changed code, and whether the decisive check existed before the change | Tests edited in the same pull request, assertions loosened, snapshots regenerated | | 4 | **Runtime signals** | Does the running software behave as specified? | Preview or staging runs, browser screenshots, traces, then canary metrics and error rates after deploy | No run at all, or a screenshot of the happy path only | ### Start with the spec delta The spec delta is the shortest artifact and the one most worth your time. It states, in sentences, which behaviour changed: "`GET /orders` now returns 50 items per page and a `next_cursor`; the old `page` parameter returns 400." You compare it with what you asked for. Scope creep, a misread requirement, and a silent contract change all show up here in seconds, and none of them needs code to spot. If nobody wrote down what was asked, write that first; see [writing acceptance criteria an agent cannot misread](/en/workflows/plan/acceptance-criteria/). ### Then map acceptance criteria to checks Every criterion in the spec gets exactly one line: the criterion, the check that proves it, and the result. "Old `page` parameter returns 400 → `orders.contract.test.ts:88` → pass." A criterion without a line is unverified, however green the pipeline looks. An agent can produce this mapping cheaply because it knows both the criteria and the tests it wrote; that is also why the next check, oracle strength, matters. ### Then ask whether the oracle could have failed The oracle is the set of checks that decides "done". Because the agent may have written both the feature and its tests, green means little until you know two things. First, **did the pull request change any test, fixture, snapshot, CI step, or lint rule?** If so, read that part of the diff; it is the only part that can make every other check lie. Second, **would the tests fail if the code were wrong?** Mutation testing answers that mechanically: it breaks the changed code on purpose and counts how many mutants the tests catch. Scope it to the changed files, because a whole-project run is slow in CI. For JavaScript and TypeScript (`@stryker-mutator/core` 10.0.0 on npm), pass the changed source files to `--mutate`: `npx stryker run --mutate "$(git diff --name-only --diff-filter=d main -- 'src/*.ts' ':!*.test.ts' | paste -sd, -)"`, and skip the step when the list is empty. For Python (mutmut 3.8.0 on PyPI), `mutmut run` takes no path arguments, so list the changed files under `only_mutate` in `[tool.mutmut]` in `pyproject.toml` (or `[mutmut]` in `setup.cfg`) before you run it; `source_paths` stays your package root (it replaced `paths_to_mutate` in mutmut 3.6.0). The full method is on [how strong your oracle is](/en/workflows/test/oracle-strength/), and keeping the agent's hands off it is on [protecting the oracle](/en/workflows/test/protect-the-oracle/). Stripe shows why a pre-existing oracle matters. Its Minions agents produce "over 1,300 Stripe pull requests" a week that are "human-reviewed, but containing no human-written code" (Stripe engineering blog, 19 February 2026). They run against an existing suite of "over three million" tests, bounded to "at most two rounds of CI" (Part 1 of the same series, 9 February 2026). The oracle predates the agent, and that is why the review can be short. ### Finish with runtime signals Tests prove what someone thought to test. A running build shows what the change does. Before merge, that means a preview or staging run with a screenshot or trace of each acceptance path. The [`agent-browser` skill](/en/ecosystem/skills/agent-browser/) lets the agent drive the running app and capture that evidence itself. After merge, it means canary metrics, error rates, and an automatic rollback. [Progressive delivery](/en/workflows/ship/progressive-delivery/) is the safety net that makes pre-merge reading optional for low-risk classes. The four artifacts together form the [evidence bundle](/en/workflows/ship/evidence-bundle/), the pull request contract that CI enforces. This page covers the reader's side; the bundle page covers the template and the CI check that fails an incomplete one. ### What counts as reading the evidence? Anthropic's article names what its engineers read in place of a diff: "Test output, the command that ran and what it returned, a screenshot, the reviewer's findings." On this page's four artifacts, the command and its result and the test output are acceptance results, the screenshot is a runtime signal, and the reviewer's findings are a second reader's report that you read instead of the diff. The article attaches one rule: "If you can't say what the checks would miss, ask for a real review rather than a quick approval." Turn the rule into a step before you approve: write one sentence that starts "These checks would miss", and decide whether you accept that gap. If you can't write it, you don't yet know what the evidence covers, so ask for a real review. The fourth prompt below asks the agent for a first draft. An agent can overlook its own gaps, so compare its answer with the escalation table before you decide. ## Which changes still need a human to read the code? Evidence works when the oracle is strong, feedback is fast, and a mistake is cheap to undo. The classes below fail at least one of those conditions, so a **named person reads the code** every time, in addition to the evidence. Keep this list short; a list that covers half the repository puts you back at Level 3. | Change class | Why evidence is not enough | Globs to start from (adjust to your repository) | | ------------ | -------------------------- | ----------------------------------------------- | | **Authentication and authorization** | Tests prove the allowed paths; the failure is a path nobody tested. The damage is a breach, not a bug. | `**/auth/**`, `**/middleware/**`, `**/*permission*`, `**/*policy*` | | **Money** | Rounding, currency, idempotency and refund edge cases surface weeks later, in customer accounts. | `**/billing/**`, `**/payments/**`, `**/pricing/**`, webhook handlers | | **Schema** | A contract other services and old clients depend on; the break shows up outside this repository's tests. | `**/schema.*`, `**/*.sql`, `**/openapi*`, `**/*.proto` | | **Data migrations** | Often irreversible, run once against production data the test fixtures do not resemble. | `**/migrations/**` | | **The oracle itself** | A change to tests, CI, lint or type config changes what "green" means for every other change. | `**/*.test.*`, `**/__snapshots__/**`, `.github/workflows/**`, lint and `tsconfig` files | | **Anything no check covers** | With no oracle, the only evidence is the code. | Decided per pull request: an acceptance line with no check | The first four classes match the ones [the human's job](/en/ladder/the-humans-job/) keeps lit on every loop. The last two are conditions, not directories: they apply wherever they occur. Route all six through `CODEOWNERS` so the required reader is enforced by the forge, not remembered by a person. The [autonomy and risk-class policy](/en/org/governance-autonomy/) is where an organization writes this list down once. ## How do you get the evidence from Claude Code, Codex and Cursor? The prompts in the next section work in all three tools. What differs is where the evidence is produced and who can block the merge. **Claude Code** Run the evidence prompt headless in CI or from your terminal. `-p` sessions start in Manual permission mode (auto mode on third-party providers or with telemetry off, from v2.1.285, where a classifier decides), so the command below passes `--permission-mode dontAsk` to deny any Bash command that would prompt on every provider (read-only commands such as `git diff` and `git log` still run). The prompt asks the agent to run the acceptance checks, so the allowlist has to include your test commands. Below, `git diff`, `git log` and the test runners are the only shell commands you allow; replace them with your repository's own: ```bash # Terminal or CI, from the repository root (Claude Code 2.1.283) claude -p "$(cat .github/prompts/evidence.md)" \ --permission-mode dontAsk \ --allowedTools "Read,Grep,Glob,Bash(git diff:*),Bash(git log:*),Bash(npm test:*),Bash(npx vitest:*),Bash(npx stryker run:*)" \ --output-format json > evidence.json ``` The run executes the pull request's own test commands with your API key in the environment. In CI, run it on `pull_request` from branches in this repository, never on `pull_request_target` with the contributor's checkout. For a second opinion on the code itself, `/code-review` reviews the local diff inside a session, and `claude ultrareview` runs the cloud-hosted multi-agent review from the shell. The managed Code Review (Team and Enterprise plans) "never approves or blocks your pull request, and the check run never concludes as a failure" (Claude Code docs, checked 2026-10-16): treat it as a reviewer's comments, not a gate. **Codex** Run the evidence prompt with plain `codex exec`, which takes it as the task and writes the final message to a file. The checks write caches and Stryker's sandbox directory, so give the run a writable workspace with the `:workspace` permission profile (beta); on a setup without profiles, `--sandbox workspace-write` does the same. Use one or the other, not both: ```bash # Terminal or CI, from the repository root (Codex CLI 0.157.1) codex exec -c default_permissions=":workspace" "$(cat .github/prompts/evidence.md)" -o evidence.md # Separate code review of the branch; review rules come from AGENTS.md codex exec review --base main -o review.md ``` `codex exec review --base main` reviews the branch against main. A custom review prompt works only without `--base`, `--commit` or `--uncommitted`: combining them fails with `error: the argument '--base ' cannot be used with '[PROMPT]'` (checked on 0.157.1). That is why the evidence prompt runs through `codex exec` and the review rules live in `AGENTS.md`. On GitHub, `@codex review` on a pull request runs the same kind of review, and it also reads its custom review rules from `AGENTS.md`. Put the escalation table's globs there so the review names the class a change falls into. **Cursor** Paste the evidence prompt into the agent in the editor before you open the pull request, and use its output as the pull request description. On the pull request, Bugbot "reviews pull requests and identifies bugs, security issues, and code quality problems". PR Routing & Approval "assigns reviewers based on code ownership and commit history, and can approve low-risk PRs when your criteria are met" (both checked on cursor.com on 2026-08-28). Encode the escalation table as the criteria: approval is allowed only when no escalation class is touched. See [PR Routing & Approval in Cursor](/en/cursor-ide/ship/rollouts-and-approval/). Runtime evidence is identical across the three tools. The `agent-browser` CLI is installed once and shared; the skill that teaches an agent to use it is installed per agent, so name each agent you use. The `-a` and `-y` flags keep `skills add` non-interactive: ```bash # Terminal. agent-browser 0.38.1 on npm, checked 2026-09-26 npm install -g agent-browser && agent-browser install npx skills add vercel-labs/agent-browser -a claude-code -a codex -a cursor -y ``` ## Copy-paste prompts for evidence-based review **Copy-paste prompt: produce the evidence for this pull request** ```text Produce the evidence for the changes on this branch against main. Do not edit any file. Output four sections, in this order, and nothing else. 1. Spec delta: the behaviour that changed, as plain sentences a product manager could check, each tied to the ticket or spec line that asked for it. List separately any behaviour change nobody asked for. 2. Acceptance results: one line per acceptance criterion, in the form criterion -> check (file:line or command) -> result. Run the checks; do not infer results. Mark any criterion without a check as UNVERIFIED. 3. Oracle changes: every test, fixture, snapshot, CI, lint or type-config file this branch modifies, with one sentence on whether the change makes a check stricter, looser, or neutral. 4. Risk class: list every file that matches auth, billing/payments/pricing, schema, migrations, or oracle (test, snapshot, CI, lint, tsconfig) paths, plus any acceptance criterion marked UNVERIFIED, and state whether a human code reader is required. If any section is empty, say so explicitly rather than omitting it. ``` **Copy-paste prompt: audit the oracle before you trust green** ```text Audit whether the tests on this branch could have caught a wrong implementation. Do not modify production code or tests. 1. List the tests that exercise the changed functions, and which of them existed on main before this branch. 2. For each changed function, describe three plausible bugs (off-by-one, wrong default, missing error branch) and name the test that would fail for each. Where no test would fail, say so. 3. Run the mutation tool this repository uses (npx stryker run for TypeScript, mutmut run for Python) on the changed files only, and report surviving mutants with file and line. 4. Flag every assertion this branch removed, loosened (toEqual -> toMatchObject, exact value -> truthy), skipped, or turned into a regenerated snapshot. End with one line: STRONG, WEAK, or NONE, and the single test that would most improve it. ``` **Copy-paste prompt: decide whether a human must read this change** ```text Classify this pull request for review routing. Read CODEOWNERS and the escalation classes in AGENTS.md or CLAUDE.md first. For each changed file, give: the change class (auth, money, schema, migration, oracle, other), the reason, and whether an existing check covers the change. Then give one verdict: - EVIDENCE: no escalation class touched and every change covered by a check. - READ: name the files a human must read and the one question they should answer while reading each file. Do not approve, merge or comment on the pull request. ``` **Copy-paste prompt: what would these checks miss?** ```text Do not edit any file. Before I approve this branch, tell me what the checks that ran on it would miss. 1. List every check that ran on this branch (tests, type check, lint, build, review findings) with its result. 2. For each behaviour this branch changes, name the check that would fail if the behaviour were wrong. Where no check would fail, write NOT COVERED. 3. End with one sentence that starts "These checks would miss" and names the likeliest gap. If you cannot name one, say what you could not inspect. ``` ## Keep a personal trust log Trust in evidence is earned per **change class**, not per tool or per agent. A trust log is a short file where you record, for each class, what you read and whether reading the code found anything the evidence missed. Keep it in your notes or in the repository under `docs/` if the team shares it. ```markdown # Trust log: Anna, orders-service | Date | PR | Class | Stage | Evidence said | Code read found | Escaped to prod? | | ---------- | ---- | --------------- | ----------- | ------------- | -------------------------- | ---------------- | | 2026-09-22 | #412 | API pagination | read-all | pass | nothing extra | no | | 2026-09-23 | #418 | API pagination | read-all | pass | unasked rename of a field | no | | 2026-09-24 | #421 | UI copy | sampled | pass | not read (sample skipped) | no | | 2026-09-25 | #425 | billing webhook | always-read | pass | missing idempotency check | no | ``` 1. **Start every class at `read-all`.** You read the evidence first, then the code, and write down whether the code showed you anything the evidence did not. 2. **Promote a class to `sampled` after a clean run.** Our starting rule is 20 consecutive pull requests in which the code read found nothing that mattered and the spec delta caught what did. The number is a team policy, not a research finding; lower it for trivial classes and raise it for anything user-facing. In `sampled`, you read the code of one pull request in five, chosen before you look at the evidence. 3. **Promote to `evidence-only` after a second clean run:** another 20 consecutive pull requests at `sampled` in which the sampled reads found nothing that mattered and no defect escaped to production in the class. You read the four artifacts and approve. 4. **Demote one stage on the first miss.** A defect that reached production, or a code read that found a real problem the evidence missed, sends the class back one stage. Fix the evidence gap first (a missing acceptance check, a weak test), then earn the stage again. 5. **Never promote the escalation classes.** Auth, money, schema, migrations and oracle changes stay at `always-read`. What improves there is the evidence you read alongside the code, not whether you read the code. The log is also your answer when someone asks why you approved a change without reading it. "This class has 40 clean pull requests and no escapes in my log" is an accountable statement. "It looked fine" is not. For rolling the same protocol out across a team, see [helping the team stop reading every diff](/en/teams/trust-transfer/). ## How does anyone learn the code if nobody reads every diff? Review has a second job besides catching defects: it spreads understanding of the code. Anthropic's article says the Claude Code team has no formal process for this. Nobody keeps a complete mental model of Claude Code in their head, and the team doesn't think anyone needs to. Someone who wants to understand a part of the codebase asks Claude to write an explainer for exactly that slice, which takes minutes and reflects the code as it is today. Some also have Claude build a small artifact that visualizes it. Working in open Slack channels adds a second source: you see what others are building while they build it. A one-line copy change teaches you almost nothing, and an explainer for the module you are about to change teaches you a lot, so read the explainer instead of trivial diffs. The prompt below adds one requirement of its own, a `file:line` citation for every claim, so you can check the explainer against the code. **Copy-paste prompt: explain the slice I am about to change** ```text Write an explainer for the part of this codebase I am about to change: how checkout applies discounts (src/billing/). Read the current code, not the docs. Cover what the pieces are and how a request moves through them, the invariants other code relies on, the tests that pin those invariants, and the three places a change is most likely to break something. Cite file:line for every claim. Do not edit any file. ``` ## What should tech leads, CTOs and executives measure? The failure the Faros data describes is not "reviews got shorter". It is 31.3% more pull requests merging with **no** review. The fix is not to demand more reading, which the same data shows does not scale. The fix is to make the lighter review a real one, and to measure that it is. Three definitions to adopt as-is: | Metric | Definition | What a bad trend means | | ------ | ---------- | ---------------------- | | **Evidence completeness** | Share of merged agent pull requests with all four artifacts present (spec delta, acceptance mapping, oracle-change statement, runtime evidence) | Approvals are running on trust, not evidence | | **Escape rate by stage** | Production defects traced to pull requests approved at `evidence-only`, divided by all pull requests approved at that stage, per change class, per month | A class was promoted too early; demote it | | **Escalation coverage** | Share of merged pull requests touching an escalation class that record a named human code reader | The escalation list is being bypassed | Report them per change class, never as one blended number. The canonical definitions and baselines live in [metrics frameworks for agentic engineering](/en/org/metrics-frameworks/). For executives, the question to ask your CTO is short: "Which classes of change merge without a human reading the code, who decided that, and what is the escape rate in those classes?" ## Signs you stopped reading too early **Tests change in the same pull request as the code they judge, and nobody notices.** This is the most common way an agent's green run becomes meaningless. Recovery: make oracle changes an escalation class in `CODEOWNERS`, and add a CI step that lists every modified test file in the pull request summary. **The evidence summary paraphrases the diff instead of reporting results.** "Updated the handler to validate input" is a description of code, not evidence. Recovery: reject any acceptance line without a `file:line` or a command and its exit status, and require the agent to run the checks it cites. **A defect escapes in a class you approve on evidence alone.** Recovery: demote the class one stage in your trust log, write the check that would have caught the defect, and add it to the oracle before you promote again. Tag the incident with the failed control, as described in [the failure taxonomy](/en/workflows/operate/failure-taxonomy/). **You can no longer explain a module you approved last month.** Anthropic's internal study (Saffron Huang and colleagues, 2 December 2025; 132 engineers and researchers surveyed, 53 interviewed) calls this the "paradox of supervision": overseeing agents needs exactly the skills that over-delegation erodes. Recovery: ask the agent for an explainer of that module (the prompt above), then read one sampled pull request a week in full, in a class you own, and ask the agent to explain the design before you read it. [Your craft and career](/en/ladder/craft-and-career/) covers which skills to keep deliberately. **The escalation list quietly grows or quietly shrinks.** A list that grows to cover half the repository sends you back to Level 3; a list that shrinks without a written decision is how auth code merges unread. Recovery: change the list only in a reviewed pull request to `CODEOWNERS`, with a reason in the commit message. **Runtime evidence is always the happy path.** One screenshot of a successful checkout proves one path. Recovery: require runtime evidence for each acceptance criterion, including the error cases the spec names. ## Where to go next with evidence-based review - [The evidence bundle](/en/workflows/ship/evidence-bundle/) — The pull request contract and the CI check that fails an incomplete one. - [Reviewing an agent's pull request](/en/workflows/ship/agent-pr-review/) — The triage protocol: spec delta, evidence, risk flags, sampled hotspots. - [How strong is your oracle?](/en/workflows/test/oracle-strength/) — Mutation score, behaviour coverage and test provenance. - [Level 4: You write the specs](/en/ladder/level-4-write-specs/) — The rung where evidence replaces diffs as the daily work. - [Helping the team stop reading every diff](/en/teams/trust-transfer/) — Shadow review, sampled review, evidence-only, per loop. - [One map of the ladder](/en/ladder/one-map/) — Where each loop sits, and the evidence each level requires. --- # Level 1–2: Assisted and Paired Coding URL: https://developertoolkit.ai/en/ladder/level-1-2-assisted-paired/ Description: Autonomy ladder Levels 1 and 2: how to tell you are at Level 2 and the three-part setup per tool (context file, plan gate, verified loop) that moves a repo up. Levels 1 and 2 of the autonomy ladder are assisted and paired coding: the agent takes discrete tasks or writes most of the change, and the developer reads every line before it is committed. Dan Shapiro estimates 90% of 'AI-native' developers sit at Level 2. Leaving Level 2 takes a context file, a plan gate and one verified task loop. This page is for developers who use Claude Code, Codex or Cursor every day and still type the same conventions into every session. Tech leads can use the five-point test to see where each repository sits. The agent is on, the diff is small, and the work still feels like yours. Nothing is broken, so nothing forces a change, and that is why so few people leave this rung. ## What you get from the Level 2 setup - A five-point test that tells you whether you are at Level 1, Level 2 or already past it. - A context file generated from your repository, with every command in it proven to exit 0. - Plan mode as the default gate: configured in Claude Code, a fixed habit before every edit in Codex and Cursor. - Two copy-paste prompts: one that writes the context file and one that runs a task to a checked stop condition. - The four ways the climb stalls, and how to recover from each. ## What do Levels 1 and 2 look like in a working day? Shapiro's article "The Five Levels: from Spicy Autocomplete to the Dark Factory" (23 January 2026) names the two rungs the coding intern and the junior developer. The difference is who holds the important work. At Level 1, in Shapiro's words, "You're writing the important stuff, but you offload specific, discrete tasks to your AI intern." A regex, a test fixture, a migration script, a component you have written 20 times. You still own the shape of the change. Level 1, like Level 0 below it, sits at the light end of [AI-assisted coding](/en/glossary/ai-assisted-coding/), the umbrella name for any use of AI to help write software. Level 2 hands over the writing and keeps the reading. The agent works through a task while you watch, and you catch the wrong turn during the run rather than in review. On this ladder, Level 2 matches [AI pair programming](/en/glossary/ai-pair-programming/), the phrase GitHub Copilot popularised in 2021. Shapiro puts this rung at "where 90% of 'AI-native' developers are living right now", and he adds the warning worth remembering: "level 2, and every level after it, feels like you are done. But you are not done." Below both sits Level 0, which few developers choose any more: "not a character hits the disk without your approval." The reading is what makes Level 2 safe, and it is already thinner than most teams assume. Sonar's survey of over 1,100 professional developers, published 8 January 2026, found that "96% of developers do not fully trust AI-generated code, and only 48% always verify it before committing." About half of the respondents do not always run the one check that Level 2 depends on. ## Are you at Level 2? The five-point test 1. **The agent gets whole tasks, not completions.** You describe an outcome, such as adding pagination to the orders endpoint, instead of accepting the end of a line you were typing. 2. **You are present for the whole run.** The session is a conversation you watch, and you stop it when it drifts. 3. **You read every line before it is committed.** You read the diff, not the summary, and you would notice a renamed variable. 4. **You are the agent's memory of the project.** You type conventions, commands and forbidden paths again in each session. 5. **Nothing runs while you are away.** No scheduled run, no second agent in a second checkout, and no job that opens a pull request on its own. Five yeses is Level 2. If item 1 is a no, you are at Level 1. Items 4 and 5 are the ones that keep you on this rung, and item 4 has a fix you can finish today: Step 1 below. ## How do you set up a repository to leave Level 2? Three changes, in this order. None of them is a new tool, and each one produces something you can check. ### Step 1: Give the agent a context file it loads every session The context file ends item 4 of the test: you stop being the agent's memory. Generate it from the repository, not from memory, and keep it to what is true today: the commands that must pass, the layout, the conventions reviewers enforce, and the paths the agent must not touch. For the layering rules and what belongs in each file, see [layered context with CLAUDE.md and AGENTS.md](/en/developer-scorecard-guide/claude-md-context/). **Claude Code** Run `/init` in the repository to generate a starter `CLAUDE.md`, then `/memory` to edit it and to turn auto memory on or off. Claude Code also reads `AGENTS.md` when the project has no `CLAUDE.md`: from v2.1.277 in first-party sessions and from v2.1.281 on Bedrock, Google Cloud and Microsoft Foundry, on both release channels since `stable` reached v2.1.287 (checked 2026-10-12). So a team that shares one file with Codex can keep a single `AGENTS.md`. **Codex** Run `/init` in the Codex prompt to create an `AGENTS.md` file. Codex "reads `AGENTS.md` files before doing any work" (OpenAI's Codex docs, checked 28 August 2026). Since codex-cli 0.150.0, an untrusted project does not supply project-level `AGENTS.md` instructions, so the file is silently ignored until you trust the repository. Accept the trust prompt Codex shows when you first open it; the [Codex quick start](/en/codex/quick-start/) covers the trust setting. **Cursor** Write the context as a project rule. Cursor's rules "provide system-level instructions to Agent" (cursor.com/docs/rules, checked 28 August 2026). Check the current rule file format against the [Cursor quick start](/en/cursor-ide/quick-start/) before you commit it. **Copy-paste prompt: write the context file from the repository** ```text Read this repository and write the context file you load every session (CLAUDE.md for Claude Code, AGENTS.md for Codex, a project rule for Cursor). Use only what is in the repo. Include: - the package scripts that actually run, and roughly how long each one takes; - the test, typecheck and lint commands; - the directory layout, one line per top-level folder; - the three conventions a new contributor gets wrong, found by reading the last 50 commits and their review comments; - the paths an agent must never edit (generated files, migrations already applied, vendored code, secrets). No aspirational rules, no generic advice, nothing you cannot point at a file for. Finish with a "Verify before you finish" block listing the exact commands that must exit 0. Then run each of those commands now and report the exit code of each one. If any command fails, fix the file, not the repository. ``` **How you know it worked:** every command in the "Verify before you finish" block exits 0 on a clean checkout, and a fresh session answers "how do I run the tests here?" correctly without you typing anything. ### Step 2: Make plan mode the default gate Reading a plan is far cheaper than reading a wrong diff. Plan mode makes the agent research and propose before it edits, so you catch the wrong approach before any code exists. **Claude Code** Press Shift+Tab to cycle to plan mode, or prefix a prompt with `/plan`. To start every terminal session in this project in plan mode, add this to `.claude/settings.json`: ```json { "permissions": { "defaultMode": "plan" } } ``` When the plan is ready, choose **Yes, manually approve edits** if you want to approve each edit as it happens. With v2.1.283 or later (both release channels on 2026-10-12), auto mode is the built-in starting mode for interactive terminal and VS Code sessions on every plan and provider (Pro, Max and Team since 14 August 2026). `claude -p` (except on third-party providers or with telemetry off, where it starts in auto mode from v2.1.285), sessions whose settings set `disableAutoMode`, and models auto mode does not support start in Manual. A `defaultMode` in your settings, like the `plan` above, replaces the built-in start. In auto mode a classifier, not you, reviews each action. That is fine for Level 2 as long as you still read the diff before you commit. To keep per-action approval instead, start with `claude --permission-mode manual`. **Codex** Type `/plan` in the Codex prompt before a change of more than a few lines. To keep edits inside the repository and let the model ask before it goes further, start the session with: ```bash codex --sandbox workspace-write --ask-for-approval on-request ``` Codex writes to the working tree, so your Level 2 read happens at `git diff` before you commit. For permission profiles and sandbox choices, see [permissions and sandboxing for agents](/en/workflows/harness/permissions-and-sandboxing/). **Cursor** Switch the agent to Plan Mode, which "creates detailed implementation plans before writing any code" (cursor.com/docs/agent/plan-mode, checked 28 August 2026). Edit the plan until each step names the files it touches and the check that proves it, and only then let the agent build. For how the plan becomes the contract for the build, see [the build stage of the lifecycle](/en/method/lifecycle/build/). ### Step 3: Run one task class end to end with a stop condition Pick a task class you repeat: a CRUD endpoint, a migration, or a bug with a reproduction. Run it once with the context file and plan mode in place, and write the acceptance criteria before the agent starts. The agent's job ends at a stop condition it can check itself, not when it says it is done. That verified path is what you later hand to an agent you are not watching. **Copy-paste prompt: a paired task with a checked stop condition** ```text Task: add cursor-based pagination to GET /orders (query params: limit, cursor; default limit 50, max 200). Before you edit anything, write a plan that lists: 1. every file you will change and why; 2. the acceptance criteria as testable statements, including limit=0, limit=201, an invalid cursor, and the last page; 3. the tests you will add for each criterion. Stop and wait for my approval of the plan. After approval, implement it. You are done only when: - each acceptance criterion has a test that fails without your change and passes with it (show me both runs); - the test, typecheck and lint commands in the context file all exit 0; - you have not edited any existing test, fixture or CI file. Finish with a short evidence summary: criterion -> test name -> result, plus anything you were unsure about. Do not summarise the diff. ``` ## How do you check the agent's work without reading every line? At Level 2 you still read every line; that is the definition of the rung. Kent Beck's [augmented coding](/en/glossary/augmented-coding/) keeps that same care for the code while the agent types. The point of the setup is to build, next to your reading, the checks that will replace it at the next level. Treat each run as a trial of those checks: | Check | Where it comes from | What it proves | | --- | --- | --- | | Command gate | "Verify before you finish" block in the context file | The change builds, types and passes the existing suite | | Acceptance tests | Criteria written in the plan, before the code | The new behaviour exists, and the test could have failed | | Protected oracle | "Do not edit existing tests" in the prompt | The agent did not make the check easier to pass | | Evidence summary | Last lines of the agent's run | Which criterion maps to which passing test | Then compare the evidence with your own read of the diff. Each time your reading finds a problem the checks missed, add a check for it. When a task class goes several runs without your reading adding anything, that class is ready for Level 3, where the agent's work reaches you as a diff you did not watch being written. You still sign off every commit at Level 2; the checks earn the right to take over later. For what a human reads once the diff is no longer the unit of review, see [reading evidence instead of code](/en/ladder/evidence-not-diffs/). ## Why the climb out of Level 2 stalls, and how to recover **The context file lists rules the repository does not follow** **Symptom:** the agent obeys the file, CI rejects the result, and the file gets the blame. **Recovery:** delete every rule you cannot point at a file or a commit for, rerun the context-file prompt above, and commit the file only after each listed command exits 0 on a clean checkout. **Reading quietly becomes skimming** **Symptom:** you approve on the agent's summary, not the diff. Sonar's 48% figure describes this failure. **Recovery:** for each task class you skim, add the command gate and acceptance tests from the table above before you skim again. Dropping the read counts as progress only when a check has taken its place. **One agent, one terminal** **Symptom:** your calendar is the throughput limit, because every session needs you in it. **Recovery:** take the task class you verified in Step 3 and run it in a second checkout while you do something else. In Codex, `codex --worktree` (or `/worktree` inside a session) gives that run its own managed Git worktree; with any tool, `git worktree add ../repo-task2` creates a second checkout. [The Level 3 guide](/en/ladder/level-3-review-diffs/) covers worktrees and parallel sessions. **The tool moved under you** **Symptom:** a flag or mode in your notes no longer behaves the way you wrote it down. In Claude Code v2.1.283, for example, auto mode became the starting mode for interactive sessions on every provider, not only on Pro, Max and Team, except on models auto mode does not support, such as the default Microsoft Foundry model. **Recovery:** run `claude --help` or `codex --help`, check your tool's quick start, and fix the context file before you automate anything on top of it. ## What should a tech lead check across a team at Level 2? A team is not at one level; each repeatable loop is. For each repository, check three things: whether a context file is committed and its commands pass in CI, whether plan mode is the configured default, and whether at least one task class has written acceptance criteria and a stop condition. A repository with all three is ready for its first unattended loop. To place teams and loops on one grid, see [one map for the ladder, the lifecycle and the factory stations](/en/ladder/one-map/). ## Where to go next from Level 2 - [Level 3: You Review the Diffs](/en/ladder/level-3-review-diffs/) — Parallel agents, review automation, and the review ceiling they hit. - [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) — What a human reads once the diff stops being the unit of review. - [Developer track](/en/start/developer/) — The reading order from this rung to running agents unattended. - [The autonomy ladder](/en/ladder/) — All six levels, and a test to place each of your loops. - [Claude Code quick start](/en/claude-code/quick-start/) — Configuration, MCP and the terminal workflow. - [Codex quick start](/en/codex/quick-start/) — Setup across the app, CLI, IDE and cloud. - [Cursor quick start](/en/cursor-ide/quick-start/) — Rules, MCP and model selection in one sitting. - [Find your level](/en/scorecard/) — Twenty-five questions that place your workflow on the ladder. --- # Level 3: You Review the Diffs URL: https://developertoolkit.ai/en/ladder/level-3-review-diffs/ Description: Level 3 of the autonomy ladder: agents write in parallel and you review every diff. How to run it, why review bandwidth caps it, and how to move to evidence. Level 3 of the autonomy ladder is the stage where agents write most of the code in parallel sessions and a developer reviews every diff before it merges. It is a stage, not a destination: review bandwidth caps it. Teams leave it by moving one change class at a time from line-by-line reading to evidence and automated gates. This page is for developers who already run two or more agents at once, and for tech leads whose review queue grows faster than the team. Five agents are running and four of them are right. You find out which four by reading, and reading is the one part of the pipeline that did not get faster. CTOs and executives can skip to the measures section: it defines the four numbers that show whether your teams have hit the ceiling. ## What you get from this Level 3 guide - A four-point test that tells you whether you are at Level 3 or still at Level 2. - The parallel-agent setup in Claude Code, Codex and Cursor, with the commands to start it. - A map of what each tool's review automation does, and the one detail that decides whether it gates anything. - Six steps that raise the review ceiling, including a CI check that turns Claude Code's review into a blocking gate. - Three copy-paste prompts: measure your review queue, make agents hand you evidence, and design the review gate. - The exit criteria for moving a change class to [reading evidence instead of code](/en/ladder/evidence-not-diffs/). ## Are you at Level 3? The four-point test Dan Shapiro's ladder (*The Five Levels: from Spicy Autocomplete to the Dark Factory*, 23 January 2026) describes Level 3 in three words: "Your life is diffs." He adds that "almost everyone tops out here." You are at Level 3 when all four statements are true: 1. **The agent writes most of each change.** You edit its output, but you rarely type the first draft. 2. **More than one agent works at a time**, each in its own checkout, while you do something else. 3. **You read every diff before it merges.** Reading, not the agent, is the step that decides. 4. **Your queue of unread work grows during the day.** Agents finish faster than you review. If statement 2 is false, you are at Level 2 and the [Level 1–2 guide](/en/ladder/level-1-2-assisted-paired/) comes first. If statement 3 is false for some changes and nothing replaced your reading, you are not at Level 4. You are at Level 3 with the check removed, which is the most common way to fail at this rung. ## How do you run parallel agents at Level 3? The mechanical unlock is filesystem isolation: one git worktree per agent, so parallel edits never touch the same files. The second unlock is one screen that shows which sessions need you. **Claude Code** Start each task in its own worktree, in the foreground or in the background (checked with Claude Code 2.1.285): ```bash # Terminal: one isolated session per task claude --worktree feature-auth claude --worktree fix-pagination --bg # runs in the background, prints an id # Terminal: agent view lists every background session and what it needs claude agents ``` Agent view is a research preview. Inside a session, the bundled `/batch` skill splits one large change into 5 to 30 units, each run by a background subagent in its own worktree. See [agent view in Claude Code](/en/claude-code/automate/agent-view/). **Codex** Worktree support is on by default since Codex 0.156.0, but a session runs in a worktree only when you opt in (checked with codex-cli 0.157.1): ```bash # Terminal: one session per task, each in a new managed Git worktree codex --worktree # Terminal: browse every agent session on the local app-server daemon codex agents ``` In the TUI, `/worktree` does the same for a running session and `/agents` opens the command center. See [Codex worktrees](/en/codex/automate/worktrees/). **Cursor** Cursor documents worktrees as the isolation unit: "Worktrees let Agent work in isolated Git checkouts" (checked 2026-08-28; cursor.com was unreachable from the writing environment on 2026-09-26). You run and watch parallel agents from the [Agents Window](/en/cursor-ide/build/agents-window/). Check the current setup steps in Cursor's documentation before you script anything around them. Worktrees isolate files, not ports, databases, caches or a shared dev server. Give each agent its own port block and its own local state, or two green runs will disagree about the same machine. The [git worktrees guide](/en/developer-scorecard-guide/git-worktrees/) covers ports and state; [ephemeral environments](/en/workflows/harness/ephemeral-environments/) cover the rest. For fleets you start from a script, see [tmux for agent fleets](/en/ecosystem/agent-tools/tmux-for-agent-fleets/) and [herdr, the agent multiplexer](/en/ecosystem/agent-tools/herdr-agent-multiplexer/). ## Why review bandwidth is the Level 3 ceiling Generation scales with the number of agents; reading scales with the number of reviewers. Two independent datasets measured what happens in between. Faros AI's *AI Engineering Report 2026: The Acceleration Whiplash* (April 2026) covers two years of telemetry from 22,000 developers and more than 4,000 teams. Throughput rose: epics completed per developer +66.2%, task throughput per developer +33.7%, pull request merge rate per developer +16.2%. Quality and review fell behind in the same window: | Faros measure (April 2026) | Change | What it tells you | | --- | --- | --- | | Median time in review | +441.5% | The queue: how long a pull request sits in review | | Median time to first review | +156.6% | The response time: how long before anyone looks | | Pull requests merged with no review | +31.3% | Review turning into approval | | Incidents per pull request | +242.7% | What the skipped reading cost | | Bugs per developer | +54% | The same, measured per person | Faros's data is vendor telemetry from its own customer base, so read it as a direction, not a forecast for your team. DX measured the load in a different unit: median pull request size grew "from 44 lines to 72 lines per pull request between July 2025 and June 2026" (DX, Justin Reock, 17 June 2026). More diffs, bigger diffs, same reader. The [state of agentic engineering](/en/ladder/state-of-agentic-engineering-2026/) page has the full numbers with sources. ## What does review automation cover in Claude Code, Codex and Cursor? All three tools ship an automated reviewer. None of them is a merge gate until you make it one. | | Claude Code | Codex | Cursor | | --- | --- | --- | --- | | Local review | `/code-review` in a session (`/review` is an alias; `--fix`, `--comment`) | `/review` in the TUI; `codex review --base main` or `codex exec review` from a script | Not verified here (cursor.com unreachable on 2026-09-26) | | Deep or security review | `claude ultrareview [target]` or `/code-review ultra`: a cloud fleet in which "every reported finding is independently reproduced and verified", typically 15 to 25 minutes (checked 2026-10-16) | `@codex security review` on the pull request (checked 2026-08-28) | Security Agents for the vulnerability pass (checked 2026-08-28) | | Review on the pull request | Managed Code Review: Team and Enterprise, generally available per Anthropic (checked 2026-10-16), tuned by a root `REVIEW.md` | `@codex review` on GitHub and GitLab, automatic reviews, rules in `AGENTS.md` (checked 2026-08-28) | Bugbot "reviews pull requests and identifies bugs, security issues, and code quality problems" (checked 2026-08-28) | | Can it block a merge? | Not by itself: "A review never approves or blocks the pull request", and the check run concludes as neutral or skipped, never as a failure (checked 2026-10-16) | Not documented (checked 2026-08-28) | PR Routing & Approval "can approve low-risk PRs when your criteria are met" (checked 2026-08-28) | Claude Code rows were checked against Anthropic's documentation and `claude --help` on 2026-09-26, except the Ultrareview duration and the Code Review rows, checked on 2026-10-16; the Codex CLI rows against `codex --help` 0.157.1. Costs for Ultrareview and Code Review are billed as usage credits; see [automated code reviews in Claude Code](/en/claude-code/ship/review-automation/) for the current figures and plan limits. The per-tool guides for [Codex review](/en/codex/ship/review-strategies/) and [Cursor Bugbot](/en/cursor-ide/ship/bugbot/) cover setup. An automated reviewer that cannot block is a second opinion. It raises your ceiling only after you decide which of its findings fail the build. ## How do you raise the review ceiling at Level 3? Six steps, in this order. Each one removes a category of work from the human queue before the next one starts. 1. **Measure the queue.** Record median time to first review, median time in review, and median pull request size per repository for the last 50 to 100 merged pull requests. Use the first prompt below. Without a baseline, you cannot tell whether any later step helped. 2. **Make the deterministic gate block.** Types, tests, lint, a schema check, and a grep for the pattern you are migrating away from all exit non-zero on failure. They cost nothing to run and cannot be talked round. Mark them as required checks, so no pull request reaches a human while one is red. 3. **Let the automated reviewer fail the build on a short list.** Pick the finding classes you trust, such as Important findings in Claude Code Review, and block on those only. Claude Code writes a machine-readable severity count into the check run, which a CI step can read with a token that has read access to checks: ```bash # CI step, after the "Claude Code Review" check run completes CHECK_RUN_ID=$(gh api "repos/$OWNER/$REPO/commits/$SHA/check-runs" \ --jq '.check_runs[] | select(.name=="Claude Code Review") | .id') counts=$(gh api "repos/$OWNER/$REPO/check-runs/$CHECK_RUN_ID" \ --jq '.output.text | split("bughunter-severity: ")[1] | split(" -->")[0] | fromjson') important=$(echo "$counts" | jq -e '.normal') || { echo 'No severity counts in check run'; exit 1; } if [ "$important" != "0" ]; then echo "Claude Code Review reported $important Important findings" exit 1 fi ``` The first call finds the ID of the `Claude Code Review` check run on commit `$SHA`, as Anthropic's Code Review page describes. The `normal` key holds the count of Important findings. `jq -e` makes the step fail closed: if the check run carries no severity counts, the build stops instead of passing. 4. **Make agents hand you evidence, not a summary of the diff.** Every agent pull request states the behaviour it changes, maps each acceptance criterion to a check that ran, and lists any test, fixture or CI file it touched. Use the second prompt below in your context file or task template. 5. **Split the queue by risk class.** Authentication and authorization, money, schema, data migrations, and any change that edits its own tests always get a human reader. Everything else can be triaged on evidence. The [agent PR review triage](/en/workflows/ship/agent-pr-review/) turns this into a six-step protocol with verdicts. 6. **Move one change class to evidence review.** Choose the class with the strongest checks, such as dependency bumps or copy changes, and approve it on the evidence from step 4 while you keep reading everything else. The exit criteria below say when a class is ready. Who signs off stays the same throughout: a named reviewer approves every merge. What changes is what that person reads. At Level 3 it is the diff; after each class moves, it is the evidence plus the diff only for the escalation classes. ## Copy-paste prompts for the Level 3 review ceiling **Copy-paste prompt: measure your review ceiling** ```text Measure this repository's review queue. Use the GitHub CLI: gh pr list --state merged --limit 100 --json number,author,createdAt,mergedAt,additions,deletions,reviews,body For each pull request compute: time to first review (createdAt to the earliest review's submittedAt), time in review (createdAt to mergedAt), size (additions + deletions), and whether it merged with zero reviews. Mark a PR as agent-authored when its author is a bot account or its body names an agent. Output one table with the median of each measure, split into agent-authored and human-authored, and into size buckets under 100, 100-400 and over 400 lines. Then list the five PRs that waited longest for a first review. Do not estimate: if a field is missing, say so and leave the cell empty. ``` **Copy-paste prompt: an evidence summary on every agent pull request** ```text Before you open the pull request, write its description in this order and nothing else: 1. Behaviour change: two sentences on what a user or caller now sees that they did not see before. No file-by-file summary of the diff. 2. Acceptance evidence: one line per acceptance criterion, as criterion -> command or test name -> result. Run every command yourself; paste the exit code, not your expectation. 3. Oracle changes: every test, fixture, snapshot, lint or CI file you added, edited or deleted, with one line on why. Write "none" if there are none. 4. Risk flags: say yes or no to each of authentication, authorization, money, schema, data migration, public API. For each yes, name the file. 5. Unsure: anything you guessed, skipped or could not verify. If any command in section 2 failed, do not open the pull request; fix it or stop and report. ``` **Copy-paste prompt: design the review gate for a repo agents now write** ```text Design the review gate for this repository, given that agents open most of the pull requests. Read .github/workflows, CODEOWNERS, the test scripts and the last 50 merged PRs first, and then output exactly four things. 1. The classes of change that may merge on a green pipeline alone, each with the command that proves it and the file globs that identify it. 2. The classes that always need a human reader, with the same globs, and one sentence each on the blast radius that earns them a human. 3. The automated review layer in between: which tool runs, on what diff size, what it may block on, and what it may only comment on. 4. The single metric that tells us the gate is failing: name it, say where it is measured today, and give the threshold that triggers a change. Propose nothing you cannot enforce with a file in this repository. ``` ## What should tech leads, CTOs and executives measure at Level 3? Four measures, per repository, show whether a team has hit the ceiling. Adopt the definitions as written and set each threshold from your own last-quarter baseline, not from another company's numbers. | Measure | Definition | Ceiling signal | | --- | --- | --- | | Time to first review | Median hours from pull request opened to first review submitted | Rises while agent count rises | | Time in review | Median hours from opened to merged | Rises faster than merge rate | | Unreviewed merge share | Merged pull requests with zero reviews, divided by all merged | Any rise that no written policy allows | | Median pull request size | Median of additions plus deletions | Grows beyond what one reviewer reads in one sitting | A rising unreviewed merge share is the number to watch. It means reviewers have stopped reading without a gate taking their place. The [review queue guide for teams](/en/teams/review-queue/) covers how to staff and route the queue. ## What breaks when five agents share one reviewer? **The queue grows before anyone notices** **Symptom:** each pull request looks reasonable, yet time to first review doubles over a month. **Recovery:** run the measurement prompt every two weeks, cap concurrent agents per reviewer at the level where the queue stays flat, and spend the freed time on step 2 of the list above. **Review turns into rubber-stamping** **Symptom:** approvals arrive minutes after a 400-line pull request opens. **Recovery:** name the classes that may merge on evidence (step 6) and require a real read for the rest. An honest gate says which classes merge unread; a reviewer who stopped reading and said nothing is the failure Faros measured. **Everyone scrolls past the automated reviewer** **Symptom:** the bot comments on every pull request and nobody acts on its findings. **Recovery:** block on one finding class (step 3), turn the rest into comments, and tune the reviewer's instructions (a root `REVIEW.md` for Claude Code Review, review rules in `AGENTS.md` for Codex) until its blocking findings are right most of the time. **Parallel agents collide outside the repository** **Symptom:** two worktrees pass alone and fail together, or one agent's migration breaks another's tests. **Recovery:** assign each worktree its own port block and database, and start dev servers on those ports instead of the defaults. **The diff is too big to review honestly** **Symptom:** a pull request exceeds what one person can hold, or Ultrareview refuses it as too large. **Recovery:** return it and split the task: one agent task, one concern, one pull request. A faster reader is not the fix. ## When is a change class ready to leave Level 3? Leave one class at a time, never the whole repository at once. A change class is ready for evidence review when all four hold: - A deterministic gate for the class is a required check and has blocked at least one bad change. - Agents cannot weaken the oracle unnoticed: any edit to tests, fixtures or CI in the same pull request sends it to a human. - Every pull request in the class carries the evidence summary from the second prompt. - Your reading over a run of recent pull requests in the class (for example, the last 10) found nothing the checks had missed. Keep a short trust log of each class, when it moved, and what sent it back. [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) explains what you read instead of the diff and the signs that you stopped reading too early. [Reviewing an agent's pull request](/en/workflows/ship/agent-pr-review/) is the day-to-day triage. When most of your merged changes are approved on evidence, your runs can last hours, and [Level 4](/en/ladder/level-4-write-specs/) begins. ## Where to go next from Level 3 - [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) — What a human reads once the diff stops being the unit of review. - [Reviewing an agent's pull request](/en/workflows/ship/agent-pr-review/) — A six-step triage: approve, return or escalate without reading every line. - [Level 4: You Write the Specs](/en/ladder/level-4-write-specs/) — Stop conditions, hooks and runs you can leave. - [Layered PR review](/en/developer-scorecard-guide/layered-pr-review/) — The three-layer gate, and why agent pull requests need it. - [Team PR review automation](/en/cto-scorecard-guide/team-pr-review-automation/) — Layered AI reviewers across a team. - [One map](/en/ladder/one-map/) — The ladder, the lifecycle and the factory stations on one grid. - [Find your level](/en/scorecard/) — Twenty-five questions that place your workflow on the ladder. --- # Level 4: You Write the Specs URL: https://developertoolkit.ai/en/ladder/level-4-write-specs/ Description: Level 4 of the autonomy ladder: you write the spec, the stop condition and the oracle, then the agent runs for hours. Setup for Claude Code, Codex and Cursor. Level 4 of the autonomy ladder is the stage where a developer writes the specification, a machine-checkable stop condition and the tests that decide it, and the agent then runs unattended for hours. The human judges the result by its evidence, not by reading every diff. Level 4 holds only while the agent cannot edit the checks that define done. This page is for developers who already run agents in parallel at [Level 3](/en/ladder/level-3-review-diffs/) and want runs they can leave, and for tech leads who decide which work may run that way. You start a large migration at six in the evening. At eight the next morning the agent reports "all tests pass" — and one of the passing tests is the one it rewrote at 3 a.m. Level 4 is the set of habits that makes that morning impossible. CTOs and executives can go straight to [the measures section](#what-should-tech-leads-ctos-and-executives-measure-at-level-4). ## What you get from this Level 4 guide - A four-point test that tells you whether you are at Level 4 or at Level 3 with the check removed. - The four artefacts that exist before a long run starts, and the order to write them in. - How to start a goal-directed run in Claude Code, Codex and Cursor, with the commands. - A Stop hook that refuses to finish while the tests are red or the test files have changed. - Two copy-paste prompts: draft and attack a stop condition, and produce the morning evidence report. - The morning review: how to accept or reject a twelve-hour run without reading its whole diff. ## Are you at Level 4? The four-point test Dan Shapiro's ladder (*The Five Levels: from Spicy Autocomplete to the Dark Factory*, 23 January 2026) describes this rung plainly: you "write a spec", then "leave for 12 hours, and check to see if the tests pass". You are at Level 4 when all four statements are true: 1. **You write the spec and the finish line, not the code.** Your working output is a spec and a condition. 2. **Runs last longer than you watch them.** An hour or more passes without a prompt from you. 3. **A check you trust decides done.** Tests, types and scripts decide, not the agent's own summary. 4. **The run cannot rewrite that check.** Test files, CI config and the hook scripts are out of its reach. If statement 4 is false, the run grades its own homework. That is Level 3 without the reviewer, the most common way this rung fails. The [one map](/en/ladder/one-map/) shows how the ladder relates to the lifecycle stages and factory stations. ## What four artefacts exist before a Level 4 run starts? None of them is code. Write them in this order, because each one is checked against the previous one. 1. **The spec.** The outcome, the constraints and what is out of scope, in a file the run reads (`spec.md`, an issue, a PRD). [Spec-driven development](/en/workflows/plan/spec-driven-development/) covers keeping it authoritative, and [acceptance criteria](/en/workflows/plan/acceptance-criteria/) covers turning each requirement into a check. 2. **The plan.** The agent writes it in plan mode, and you read it. A plan is a page of text, not a 3,000-line diff: the cheapest review of the whole run. See [plan mode across tools](/en/method/lifecycle/build/) and Cursor's [PRD workflow](/en/cursor-ide/quick-start/prd-workflow/). [AI-DLC](/en/glossary/ai-dlc/) formalises this step: the AI drafts a plan and implements only after the team validates it. 3. **The oracle.** The tests, types and scripts that decide whether the spec holds, written or reviewed by you **before** the run. Tests written first are the only part of the oracle the agent did not author, which is the Level 4 case for [test-driven development](/en/method/principles/test-driven-development/): not purity, provenance. [Oracle strength](/en/workflows/test/oracle-strength/) covers whether those checks can catch a wrong answer at all. 4. **The stop condition.** One finish line stated as commands that must exit 0, plus a round or turn budget. The prompt below drafts it and then attacks it. **Copy-paste prompt: draft a stop condition, then attack it** ```text Read spec.md and the plan we agreed. Draft the goal text for this run as five blocks and nothing else. Objective: one sentence, one finish line. Read first: the exact files that define correct, including one already-correct example in this repository. Done when: the commands that must exit 0 (tests, typecheck, lint), plus one grep proving the old pattern is gone. Say that each command's output must appear in the transcript. Never touch: the test files that make up the oracle, listed by path, plus CI config, .claude/run-base, and .claude/ or .codex/ settings. Budget: one line appended to RUN_LOG.md per round, and "or stop after 30 turns and report what is left" as the last clause. Then attack your own draft. List every way the condition could be satisfied by weakening a test, deleting a case, skipping a suite, or claiming a result without running the command. Rewrite the wording so none of them works, and print only the final version. ``` Stripe's in-house agents show what a bounded condition looks like at scale: "at most two rounds of CI", after which the branch goes back to a human (Stripe engineering blog, 9 February 2026). A budget is part of the condition, not an afterthought. ## How do you start a goal-directed run in Claude Code, Codex and Cursor? All three tools have a `/goal` command. They differ in who checks the condition and what else you can attach to it. **Claude Code** Checked against Claude Code 2.1.283 (`latest` channel) and its `/goal` documentation on 26 September 2026. ```text # In the session: plan first, approve the plan, leave plan mode (Shift+Tab to auto mode) /plan # then set the goal; it starts a turn at once /goal all tests in test/billing pass, `npm run typecheck` exits 0, no file under test/ is modified, or stop after 30 turns /goal # status: condition, turns, token spend, last verdict /goal clear # stop early ``` After each turn, a small, fast model (Haiku by default on the Claude API) judges whether the condition holds. It returns "not yet met", "met" or "impossible". The condition can be up to 4,000 characters, and one goal is active per session. A goal does not change your permission mode, so a goal set while plan mode is still on cannot edit anything. Switch to auto mode first to let turns continue unattended. The evaluator "doesn't run commands or read files independently". It reads the transcript. That is why the condition above names the commands, and why the Stop hook in the next section runs them itself. For a run from a script or CI, headless mode runs the whole loop in one call. A `claude -p` session starts in Manual mode, not auto (v2.1.283; auto mode on third-party providers or with telemetry off, from v2.1.285), so pass the permission mode explicitly or the run stalls on its first unapproved tool call: ```bash # Terminal: auto mode for unattended turns; stream each turn so a long goal does not look stuck claude -p "/goal CHANGELOG.md has an entry for every PR merged this week" \ --permission-mode auto --output-format stream-json --verbose ``` For a tighter run, replace `--permission-mode auto` with `--allowedTools` and list only the commands the goal needs. **Codex** Checked against codex-cli 0.157.1 on 26 September 2026: `goals` is a stable feature and on by default, and `/goal` sets or views "the goal for a long-running task". Since 0.151.0, tokens spent by nested subagents count toward the goal's budget. ```text # In the Codex TUI: plan first, approve the plan, then set the goal /plan /goal all tests in test/billing pass, `npm run typecheck` exits 0, no file under test/ is modified, or stop after 30 turns and report what is left ``` Codex's own guidance on writing the objective (Long-running work page, checked 28 August 2026) asks for three parts: "Outcome, Constraints, Verification". Codex also has a `Stop` hook event among its 12 hook events; hooks need persisted trust before they run. Codex has no `/loop` command (absent from the developer-commands reference, the 0.162.1 slash-command enum and that binary's strings, checked 2026-10-12). **Cursor** Cursor announced `/goal` in its 19 August 2026 changelog: "Use `/goal` to give the agent a long-lived objective to work towards until it's fully complete." Its CLI slash-commands reference marked the command "Rolling out" when checked on 28 August 2026, and still does on 2026-10-12. Plan Mode "creates detailed implementation plans before writing any code", and Cursor hooks run "before or after defined stages of the agent loop and can observe, block, or modify behavior" (all checked 28 August 2026; cursor.com was unreachable from the writing environment on 26 September 2026). Check Cursor's hooks documentation for the current event names before you script a stop gate. `/loop` (a bundled skill since 3.5, 20 May 2026) re-runs a prompt on a local schedule until an outcome is reached or you stop it; `/goal` works toward one finish line without a schedule. [The /goal command](/en/workflows/automate/goal-command/) walks through each tool in depth, and [the /loop command](/en/workflows/automate/loop-command/) covers cadence-driven runs. ## How do you stop a run from finishing while the tests are red? Use a hook: a script the tool runs at a fixed point in the loop, whatever the model decides. In Claude Code, a `Stop` hook runs when the agent tries to end its turn. Exit code 2 blocks the stop and sends your stderr text back to the agent as the reason to continue. Unlike the `/goal` evaluator, the hook runs the commands itself. Before the run starts, record the commit it starts from, so the hook can compare against it however many times the agent commits: ```bash git rev-parse HEAD > .claude/run-base ``` ```bash #!/usr/bin/env bash # .claude/hooks/require-green.sh — Stop hook: no finishing while the oracle is red cd "${CLAUDE_PROJECT_DIR:-$(git rev-parse --show-toplevel)}" || exit 0 [ -f .claude/run-base ] || exit 0 # armed only during a recorded run # 1. The oracle must be untouched: the run may not edit the tests it is graded by. # Diff against the commit recorded before the run, not HEAD, so a committed # test edit still counts, and list new untracked test files too. base=$(cat .claude/run-base) changed=$( { git diff --name-only "$base" -- test/; git ls-files --others --exclude-standard -- test/; } ) if [ -n "$changed" ]; then printf 'Revert your edits to the test files and fix the code instead:\n%s\n' "$changed" >&2 exit 2 fi # 2. The oracle must be green. if ! out=$(npm test --silent 2>&1); then printf 'Tests are failing. Fix the code, not the tests:\n%s\n' "$(printf '%s' "$out" | tail -n 40)" >&2 exit 2 fi exit 0 ``` Register it in `.claude/settings.json` (make the script executable with `chmod +x`): ```json { "hooks": { "Stop": [ { "hooks": [ { "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/require-green.sh", "timeout": 600 } ] } ] } } ``` The hook arms itself only during a recorded run: without `.claude/run-base` it exits 0 silently, so ordinary sessions in the same repository see no hook errors. Delete the file when the run is reviewed. Three limits to know before you rely on it. Claude Code applies an eight-consecutive-continuation cap: after stop hooks have continued the turn eight times in a row, the next block is overridden and the turn ends (`CLAUDE_CODE_STOP_HOOK_BLOCK_CAP` raises it). The settings file lives in the repository, so the run can edit it too; list it under "Never touch" and check it in CI. And `/goal` is itself a Stop hook, so it is unavailable when `disableAllHooks` is `true`. The command tells you so instead of failing silently. The same gate in the other tools: - **Codex** runs the same script from its `Stop` hook, but do not copy the Claude Code registration. Codex 0.157.1 does not set `CLAUDE_PROJECT_DIR`, so `${CLAUDE_PROJECT_DIR}/.claude/hooks/require-green.sh` expands to `/.claude/hooks/require-green.sh`, the shell exits 127 with "not found", and the gate never runs. Codex runs the command through a shell in the session's working directory, so resolve the repository root with git in `.codex/hooks.json`: ```json { "hooks": { "Stop": [ { "hooks": [ { "type": "command", "command": "\"$(git rev-parse --show-toplevel)/.claude/hooks/require-green.sh\"", "timeout": 600 } ] } ] } } ``` Then open `/hooks` and trust it: project hooks do not run until trust is persisted, and again after anyone changes the file (Codex 0.157.1). [Agent hooks across tools](/en/developer-scorecard-guide/agent-hooks/) covers the rest of the Codex registration. - **Cursor** hooks can block the agent loop, but its event names were not verifiable from the writing environment on 26 September 2026. Check them in Cursor's hooks documentation before you port the script. [Hooks in Claude Code](/en/claude-code/automate/hooks-automation/) and [agent hooks across tools](/en/developer-scorecard-guide/agent-hooks/) cover the other events. [Protecting the oracle](/en/workflows/test/protect-the-oracle/) goes further: locked test paths, CODEOWNERS, and holdout checks kept out of the repository. ## How do you check a twelve-hour run without reading every line? You read evidence, in a fixed order, and read code only where evidence is not enough. The agent assembles the evidence; you decide. 1. **Re-run the oracle yourself**, or read the CI result on the pushed branch. Never accept the run's own report of green. 2. **Check that the oracle is intact.** `git diff --stat $(cat .claude/run-base) -- test/ .github/ .claude/` must be empty unless the spec asked for test changes. 3. **Read the spec delta.** Which acceptance criteria are met, and which ones the agent says it could not meet. 4. **Read `RUN_LOG.md`.** Look for the round where progress stalled or the approach changed. Long runs degrade: SlopCodeBench (arXiv, 25 March 2026) had 15 coding agents extend their own code over 196 checkpoints, found that "no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints", and measured the decay as structural erosion and verbosity. 5. **Read code only for the escalation classes.** Schema and migrations, auth, money, and anything a customer sees. [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) defines these classes. 6. **Sign off or return.** The developer who wrote the spec signs off on the evidence. A change in an escalation class also needs its code owner. **Copy-paste prompt: the morning evidence report** ```text The run toward the goal in RUN_LOG.md has ended. Do not change any code. Produce EVIDENCE.md with these sections and nothing else: 1. Oracle: run the full test suite, the typecheck and the lint step now, and paste each command with its exit code and last 10 lines of output. 2. Oracle integrity: the output of git diff --stat $(cat .claude/run-base) -- test/ .github/ .claude/ and one sentence on every file it lists. 3. Spec coverage: a table of every acceptance criterion in spec.md with met / not met / partly met and the test that proves it. 4. Escalation classes: every changed file that touches migrations, auth, payments or user-visible copy, with one line on what changed. 5. Turning points: the rounds in RUN_LOG.md where the approach changed or progress stalled, and why. 6. Doubts: what you are least sure of, and the check that would settle it. ``` The same EVIDENCE.md, attached to a pull request, is the [evidence bundle](/en/workflows/ship/evidence-bundle/) that reviewers read instead of the diff. ## What should tech leads, CTOs and executives measure at Level 4? Four measures, per repository or per change class, show whether Level 4 is working or whether runs only look finished. Adopt the definitions as written and set thresholds from your own baseline. | Measure | Definition | Warning signal | | --- | --- | --- | | Run acceptance rate | Unattended runs whose output merged without rework, divided by all runs started | Falls as runs get longer | | Oracle-edit rate | Runs whose diff touched test, CI or hook files without the spec asking for it | Any rise | | Escaped defects per class | Production incidents traced to changes approved on evidence, per change class | A class that keeps escaping goes back to Level 3 | | Budget overrun | Runs stopped by the turn or token budget rather than by the condition | Conditions that are unreachable or vague | The oracle-edit rate is the one to watch. A run that edits its own checks has not reached Level 4, whatever the acceptance rate says. ## What breaks when a twelve-hour run comes back green? **The agent edited the oracle** **Symptom:** the suite is green and a test file is in the diff. **Recovery:** revert the test edits, re-run the oracle on the result, and add the paths to the hook and to "Never touch". Keep the decisive check in CI, where the run cannot rewrite it. **The evaluator believed a claim** **Symptom:** the goal is "met", but nobody ran the tests in the final turn. **Recovery:** write "the output of `npm test` must appear in the transcript" into the condition, and add the Stop hook so the check runs whatever the agent says. **The run wandered or thrashed** **Symptom:** the log shows the same fix tried three ways, or work outside the spec. **Recovery:** split the objective in two, name one already-correct example to copy, and restart from the last good commit in the log, not from the end state. **The budget ran out before the goal did** **Symptom:** the quota is gone and the condition is still unmet. **Recovery:** the condition was unreachable or too broad. Add a turn clause ("or stop after 30 turns and report"), and prove the condition can pass by running it on a hand-made fix first. **The run never started** **Symptom:** `/goal` refuses to start in Claude Code. **Recovery:** read the message. The usual cause is hooks being disabled (`disableAllHooks`), managed-hooks-only policy, or an untrusted folder, since `/goal` runs through the hooks system. **Green, but wrong** **Symptom:** every check passes and the feature still misbehaves. **Recovery:** the oracle is too weak for this class. Strengthen it with a failing test for the bug, and move the class back to reading code until the new checks have caught something on their own. ## When is a change class ready for Level 5? [Level 5](/en/ladder/level-5-run-the-factory/) removes the human from the loop for whole stations, not single runs. A change class is ready when unattended runs in that class keep a high acceptance rate, the oracle-edit rate is zero, escaped defects stay flat, and the specs, not the prompts, are what your team reviews. Everything a stop condition cannot decide (intent, taste, the escalation classes) stays with people; [the human's job](/en/ladder/the-humans-job/) lists it. Compound engineering keeps each run from starting at zero: every finished run leaves a rule, a check or a skill behind. See [compound engineering](/en/ecosystem/frameworks/compound-engineering/), the packaged discipline in [Superpowers](/en/ecosystem/frameworks/superpowers/), and [grill me](/en/ecosystem/skills/grill-me/) as the alignment pass to run before a long job. ## Where to go next from Level 4 - [Level 5: You Run the Software Factory](/en/ladder/level-5-run-the-factory/) — Six stations, and what each tool documents for them. - [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) — What a human reads once the diff stops being the unit of review. - [Protecting the oracle](/en/workflows/test/protect-the-oracle/) — Checks the agent cannot edit: locked paths, CI-owned gates, holdouts. - [The /goal command](/en/workflows/automate/goal-command/) — Goal-directed autonomous runs, tool by tool. - [Spec-driven development](/en/workflows/plan/spec-driven-development/) — Keeping the spec, not the code, as the thing you review. - [Find your level](/en/scorecard/) — Twenty-five questions that place your workflow on the ladder. --- # Level 5: You Run the Software Factory URL: https://developertoolkit.ai/en/ladder/level-5-run-the-factory/ Description: Level 5 of the autonomy ladder: six factory stations mapped to Claude Code, Codex and Cursor, and the test for which loops may run unattended. Level 5 of the autonomy ladder is the software factory: a pipeline where agents pull work from a queue, build and verify it, and open pull requests that no human wrote. It has six stations (intent, harness, loop, graph, verification and release), and a loop may run unattended only when an oracle the agent cannot edit decides it is done. This page is for the tech lead or platform engineer who already runs [Level 4 loops](/en/ladder/level-4-write-specs/) and for the CTO deciding whether to fund a factory pilot. The typical situation: your `/goal` runs finish overnight, the review queue is now the slowest part of delivery, and someone asks whether the agents could merge dependency bumps on their own. This page answers which loops can, what each station needs first, and what Claude Code, Codex and Cursor ship for it. ## How close is anyone to a dark factory today? Dan Shapiro named this rung in [The Five Levels: from Spicy Autocomplete to the Dark Factory](https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/) (23 January 2026): "It's a black box that turns specs into software", where "humans are neither needed nor welcome". He places only "a handful of people" there, in teams of "less than five people". The best-documented production case, Stripe's Minions, keeps a human in the loop. Stripe reported that "over 1,300 Stripe pull requests" a week are "completely minion-produced, human-reviewed, but containing no human-written code" ([Stripe, 19 February 2026](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents-part-2)). The agents run against "over three million" existing tests and get "at most two rounds of CI" ([Stripe, 9 February 2026](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents)) before the branch goes back to a human. The proof layer predates the agents. The cost of skipping that layer is also measured. Faros AI's telemetry from 22,000 developers ([AI Engineering Report 2026, April 2026](https://www.faros.ai/research/ai-acceleration-whiplash)) shows task throughput per developer up 33.7% and PR merge rate per developer up 16.2%, while incidents per pull request rose 242.7%, median time in review rose 441.5%, and 31.3% more pull requests merged with no review. So treat Level 5 as a property of individual loops, never of a whole company. ## What are the six stations of a software factory? Each station owns one decision. A station that emits no evidence is one you supervise by reading. | Station | Decision it owns | Evidence it must emit | Symptom when it is missing | |---|---|---|---| | **Intent** | What the agent knows before it starts | A spec with acceptance criteria and a named stop condition | Plausible code that solves the wrong problem | | **Harness** | What the agent may run and reach | A permission and sandbox policy under version control | A run that works because it had more access than it should | | **Loop** | When the agent stops | A machine-checkable stop condition and a retry cap | Runs that never finish or finish by declaring victory | | **Graph** | Who does what when there are several agents | A written topology: planner, workers, judge, isolation | Agents overwriting each other's files or context | | **Verification** | What counts as proof | Deterministic checks the run cannot edit, plus review findings | Green checks on tests the agent rewrote | | **Release** | How work leaves the building | An evidence bundle on the pull request and a named merger | A merge nobody can explain a week later | The stations map onto the lifecycle and the ladder on the [one-map page](/en/ladder/one-map/). The deeper build guide, with the full autonomy ledger, is [software factories](/en/method/principles/software-factories/). ## Which loops may run unattended? A level belongs to a loop, not to a team. Four questions decide whether a loop may run dark, and a loop must pass all four: 1. **What oracle decides "done"?** Name the specific check. "The test suite and the type checker pass" is an oracle. "It looks right" is not. 2. **Can the agent fake it?** If the agent can edit the oracle, the oracle is a suggestion. Tests inside the diff fail this question by construction. 3. **How long until a wrong answer surfaces?** Minutes is a pass. "A customer notices in three weeks" is a fail. 4. **What is the blast radius if it is wrong?** Reversible and contained is a pass. A schema migration or a payment path is a fail. Adopt this table as the rule your team applies before any loop leaves supervision: | Loop class | Oracle | Agent can edit it? | Time to surface | Blast radius | Verdict | |---|---|---|---|---|---| | Patch and minor dependency updates | Full CI suite | No, if CI config and tests are protected | Minutes | One revertable commit | Runs unattended, human merges | | Lint and codemod migrations | Linter plus type checker | No, if rule config is protected | Minutes | Mechanical, revertable | Runs unattended, human merges | | Flaky-test quarantine | Re-run statistics from CI history | Yes, unless quarantine lives outside the repo | Days | Hides real failures | Supervised | | Feature work from a spec | Acceptance tests written with the spec | Yes, if they live in the same diff | Days to weeks | Product behaviour | Supervised, evidence review | | Auth, money, schema, migrations | Partial at best | Varies | Weeks | Irreversible | Stays with a human reading code | The last row is the short list of change classes where someone still reads the code; [reading evidence instead of code](/en/ladder/evidence-not-diffs/) explains why. **Copy-paste prompt: find the first loop that can run dark** ```text Read the last 60 merged pull requests in this repository (use `gh pr list --state merged --limit 60 --json number,title,files,labels`) and group them into loop classes: repeated kinds of change such as dependency updates, lint fixes, flaky tests, copy changes or feature work. For each class, answer the autonomy ledger in a markdown table: 1. the oracle that decides "done", as a command that exits 0; 2. whether an agent working on that class could edit that oracle, citing the file paths it would have to touch; 3. how long a wrong change of that class took to surface, using reverts and follow-up fixes in the git log as evidence; 4. the blast radius if it is wrong. End with the single class that passes all four, or say that none does and name the one change that would make the best candidate pass. Do not modify any files. ``` ## How do you stand up the first factory loop? Build one loop at a time, harness first. The steps use dependency updates, the table's first row, because its oracle already exists. 1. **Write the intent as a file in the repository**, for example `.factory/dep-bump.md`, so every run gets the same instructions and the prompt goes through code review like any other change. Use the prompt below as its content. 2. **Put the oracle out of the agent's reach.** Add a `CODEOWNERS` entry for `tests/`, `.github/` and the lint and type configs, and turn on "Require review from Code Owners" in the branch ruleset; a `CODEOWNERS` file on its own only requests reviewers. Make the CI checks required, and add the CI step from [the failure modes below](#where-a-software-factory-rots-first) that fails any factory branch touching protected paths. The run can still change code, but it cannot merge a weaker test. [Protecting the oracle](/en/workflows/test/protect-the-oracle/) covers hooks and holdout tests. 3. **Fix the harness.** Run the agent in a disposable CI job or cloud environment with an explicit allowlist, a spending cap and no secrets except the model credential. Check out with `persist-credentials: false` so no write token sits in `.git/config` while the agent runs. Opening the pull request is a separate job that runs no agent: the agent job uploads its changes as a patch artifact, and the second job, with its own narrowly scoped token (`contents: write`, `pull-requests: write`), applies the patch, pushes and opens the pull request. 4. **Run the loop headless** with the tool's non-interactive mode (commands below). The loop's stop condition is in the prompt: at most two fix attempts, which mirrors Stripe's two CI rounds. 5. **Release with evidence, not with a diff.** The pull request carries the run summary, the exit codes of the three checks and the dependency's changelog link. A human merges from that [evidence bundle](/en/workflows/ship/evidence-bundle/) and reads code only when the evidence is incomplete. 6. **Measure before you widen.** Track revert rate, incidents per pull request and review wait for the loop for four weeks. Add a second loop class only when those three are flat or better. **Copy-paste prompt: the build station for dependency updates** ```text You are the build station of a dependency-update loop. Scope: one outdated direct dependency per run, patch or minor versions only. If the only update available is a major version, stop and report it. 1. Run `npm outdated --json`, pick the patch or minor update that has been available longest, and update only that package and the lockfile. 2. Run `npm test`, `npm run lint` and `npm run typecheck`. If any fails, fix the application code, never the tests, the lint config or the type config. Make at most two fix attempts, then stop. 3. Do not edit anything under tests/ or .github/, or any lint or tsconfig file. 4. Finish with a JSON summary: package, from, to, each of the three commands with its exit code, the files you changed, and "status": "ready" or "needs-human" with a one-sentence reason. ``` Each headless command runs from the repository root in the CI job or disposable runner from step 3. **Claude Code** `dontAsk` mode auto-denies every tool call that would otherwise prompt, so the run never waits for input: only file reads, read-only commands and the tools on the allowlist run. ```bash claude -p "$(cat .factory/dep-bump.md)" \ --permission-mode dontAsk \ --allowedTools "Read" "Edit" "Bash(npm outdated *)" "Bash(npm install *)" \ "Bash(npm test)" "Bash(npm run lint)" "Bash(npm run typecheck)" \ --max-budget-usd 5 \ --output-format json > .factory/last-run.json ``` For scheduled runs without your own CI, a [routine](/en/claude-code/automate/routines/) runs the same prompt as a cloud session on a schedule, from an API call or from a GitHub event. Routines have no permission-mode picker and no approval prompts during a run, so the allowlist lives in the environment, not in the session. **Codex** `codex exec` needs no approval flag: it has no `-a` of its own and already runs with approval `never`, so a command outside the sandbox fails instead of waiting for input (checked on CLI 0.157.1). The `workspace-write` sandbox restricts network access unless you enable it, and this loop needs the registry. ```bash codex exec \ --sandbox workspace-write \ -c sandbox_workspace_write.network_access=true \ -o .factory/last-run.md \ "$(cat .factory/dep-bump.md)" ``` Legacy `--sandbox` still works in 0.157.1; OpenAI now prefers permission profiles (beta, `-c default_permissions=":workspace"`). Do not combine a profile with `--sandbox` until re-verified, because OpenAI says the two systems do not compose. For recurring runs, Codex [automations](/en/codex/automate/automations/) schedule the same task in the background. In CI, `openai/codex-action@v1` wraps the same non-interactive run. **Cursor** Cursor's CLI has a print mode (`-p`, `--print`) and runs in GitHub Actions, but for a factory loop the more direct route is an [Automation](/en/cursor-ide/automate/cloud-agents-automations/) with a Scheduled or Source control trigger that starts a Cloud Agent with `.factory/dep-bump.md` as its task. Cloud Agents run in isolated VMs, and the Cloud Agents API sends a `statusChange` webhook when a run finishes, so your systems learn the result without polling (Cursor documentation, checked 2026-08-28). Never add `--dangerously-skip-permissions`, `bypassPermissions` or Codex's `--dangerously-bypass-approvals-and-sandbox` to a factory loop outside a disposable, network-restricted sandbox. A loop that needs full access to pass has told you its harness is wrong. ## How do you know factory output is right without reading every diff? - **Deterministic gates decide, models advise.** Types, tests, lint and fail-closed migration checks decide whether a change may merge. Review agents such as Claude Code's `/code-review`, Codex's `/review` and Cursor's Bugbot add findings, but none of them is deterministic, so none may be the only gate. - **The oracle lives outside the diff.** Required checks come from a workflow the factory branch cannot change, either a ruleset "Require workflows to pass" pinned to the default branch or a `pull_request_target` job (a `pull_request_target` job must not check out or execute the branch's code with secrets in scope; run the branch's tests in a separate unprivileged job), and a CI step fails any factory branch that touches `.github/`. A `pull_request` workflow runs from the branch's own merge ref, so a branch that edits it changes its own check. `CODEOWNERS` with required code-owner review covers test and config paths, and a holdout suite the agent never sees catches overfitting to visible tests. - **Every pull request carries an evidence bundle:** the intent file, the commands and exit codes, the review findings and the run log. The [agent PR review](/en/workflows/ship/agent-pr-review/) page covers reading one. - **A sampling audit replaces line-by-line review.** Each week, a named owner reads a small random sample of merged factory pull requests end to end. If the sample finds a defect the gates missed, that loop goes back to supervised until a new check covers the gap. - **Four numbers per loop:** revert rate, incidents per pull request, review wait and share merged with no human approval. The loop's owner signs off on the loop, and the merger signs off on each pull request from its evidence. Nobody signs off on "the factory" as a whole; that is [the human's job](/en/ladder/the-humans-job/) of reading the seams. **Copy-paste prompt: write the verification contract for one station** ```text Station: verification. Loop: patch and minor dependency updates. Write the station's contract as a markdown table with one row each for: the inputs it accepts; the deterministic validators that must pass before hand-off, each as a command that exits 0; whether the agent can edit any of those validators, and if it can, a replacement that lives outside the diff; the failure route, naming which station receives a FAIL and what it gets back; the maximum retries before the run stops; and the human escalation trigger. Then answer the autonomy ledger for this loop: what oracle decides done, can the agent fake it, how long until a wrong answer surfaces, and what the blast radius is. End with a one-line verdict: runs unattended, or stays supervised and why. ``` ## What does each tool ship for each station? Claude Code rows, including their "Not documented" cells, were re-checked on 2026-09-26 against code.claude.com and the installed CLI (2.1.283). Codex rows were checked against CLI 0.157.1 on 2026-09-26 where the CLI shows them, and against OpenAI's documentation on 2026-08-28 otherwise. Cursor rows were last verified on 2026-08-28, because cursor.com was unreachable on 2026-09-26. "Not documented" is a gap in the documentation on that date, not proof the tool cannot do it. Re-check the vendor page on the day you build against it. ### Intent: context, procedure and plan | Capability | Claude Code | Codex | Cursor | |---|---|---|---| | Persistent project instructions | [`CLAUDE.md` and auto memory](https://code.claude.com/docs/en/memory.md) | [`AGENTS.md`](https://learn.chatgpt.com/docs/agent-configuration/agents-md.md) | [Rules](https://cursor.com/docs/rules) | | Reusable procedure | [Skills](https://code.claude.com/docs/en/skills.md) | Skills (`/skills`) | [Agent Skills](https://cursor.com/docs/skills) | | Distributable bundle | [Plugins](https://code.claude.com/docs/en/plugins.md) | Plugins (`codex plugin`) | [Plugins](https://cursor.com/docs/plugins) | | Plan before implementing | Plan mode (`/plan`) | Plan mode (`/plan`) | [Plan Mode](https://cursor.com/docs/agent/plan-mode) | | Guidance on writing a verifiable objective | [Write an effective `/goal` condition](https://code.claude.com/docs/en/goal.md): end state, stated check, constraints | [Long-running work](https://learn.chatgpt.com/docs/long-running-work.md): outcome, constraints, verification | Not documented (2026-08-28) | Both vendors describe the same three parts: the outcome, the check that proves it and what must not change. Copy them into every intent file, whichever tool runs it. ### Harness: permissions, sandbox and compute | Capability | Claude Code | Codex | Cursor | |---|---|---|---| | Permission policy | [Permission modes](https://code.claude.com/docs/en/permission-modes.md): Manual, `acceptEdits`, `plan`, `auto`, `dontAsk`, `bypassPermissions` | Approval policies `untrusted`, `on-request` (default), `granular`, `never`; permission profiles (beta) | [Run modes](https://cursor.com/docs/agent/security/run-modes) | | Approvals decided by a model | Auto mode classifier | Auto-review (`--approve-for-me`) | Auto-review for local tool calls | | Operating-system sandbox | [Sandboxed Bash](https://code.claude.com/docs/en/sandboxing.md) (macOS, Linux, WSL2) | `--sandbox` modes and `codex sandbox` | `/sandbox` in the CLI | | Lifecycle hooks | [Hooks](https://code.claude.com/docs/en/hooks.md) | Hooks (need persisted trust) | [Hooks](https://cursor.com/docs/hooks) | | External tools | [MCP](https://code.claude.com/docs/en/mcp.md) | MCP (`codex mcp`) | [MCP](https://cursor.com/docs/mcp) | | Self-hosted compute | Self-hosted environments (Team and Enterprise beta) | Not documented (2026-08-28) | Private workers in the Cloud Agents API | | Push external events into a live session | [Channels](https://code.claude.com/docs/en/channels.md) (research preview) | Not documented (2026-08-28) | Not documented (2026-08-28) | Claude Code's classifier blocks actions that escalate beyond the request, target unrecognized infrastructure or appear driven by hostile content: a judgement made by a model. For an unattended loop, prefer an explicit allowlist and treat any classifier as a second line. ### Loop: stop conditions, schedules and headless runs | Capability | Claude Code | Codex | Cursor | |---|---|---|---| | Goal or stop condition | [`/goal`](/en/workflows/automate/goal-command/) | `/goal` (stable, on by default) | `/goal` (rolling out since 2026-08-19 and still so, checked 2026-10-12) | | Interval loop | [`/loop`](/en/workflows/automate/loop-command/) | No `/loop` (checked 2026-09-26, CLI 0.157.1) | `/loop` bundled skill | | Scheduled unattended runs | [Routines](/en/claude-code/automate/routines/) (research preview, one-hour minimum) | [Automations](/en/codex/automate/automations/) | [Automations](/en/cursor-ide/automate/cloud-agents-automations/) | | Headless invocation | `claude -p` | `codex exec` | Print mode (`-p`) | | Programmatic SDK | Claude Agent SDK | Codex SDK | `@cursor/sdk`, `cursor-sdk` | | Isolated cloud run | Claude Code on the web | Codex cloud | Cloud Agents | The triggers differ more than the runners: routines start on a schedule, an API call or a GitHub event; Codex's automations named Gmail, Slack and GitHub pull request activity (2026-08-28); Cursor's automations add Webhook, Linear, Sentry and PagerDuty. If your factory pulls from a queue, that trigger list is the queue. ### Graph: many agents without collisions | Capability | Claude Code | Codex | Cursor | |---|---|---|---| | Delegated subagents | [Subagents](https://code.claude.com/docs/en/sub-agents.md), 20 running per session by default | Subagents (`/subagents`) | [Subagents](https://cursor.com/docs/subagents), on their own VMs since 2026-08-19 | | Peer agents that message each other | Agent teams (experimental, off by default) | Not documented (2026-08-28) | Not documented (2026-08-28) | | Script-driven orchestration | [Dynamic workflows](/en/claude-code/automate/dynamic-workflows/) | Not documented (2026-08-28) | Not documented (2026-08-28) | | Filesystem isolation | Worktrees (`--worktree`) | Worktrees (`--worktree`, `/worktree`) | [Worktrees](https://cursor.com/docs/configuration/worktrees) | | One screen for many agents | Agent view (`claude agents`, research preview) | `codex agents` command center | Agents window | Dynamic workflows "orchestrate many subagents from a script Claude writes and you can rerun": a factory floor you can version. [Orchestration patterns](/en/workflows/automate/orchestration-patterns/) covers the planner, workers and judge topology in all three tools. The general name for coordinating several agents like this is [agent orchestration](/en/glossary/agent-orchestration/). ### Verification: review and rollback | Capability | Claude Code | Codex | Cursor | |---|---|---|---| | Review the local diff | `/code-review` | `/review`, `codex review` | Not documented (2026-08-28) | | Deep multi-agent review | [Ultrareview](https://code.claude.com/docs/en/ultrareview.md): `/code-review ultra` or `claude ultrareview` | Not documented (2026-08-28) | Not documented (2026-08-28) | | Managed pull request review | Code Review (Team and Enterprise; generally available per Anthropic, checked 2026-10-16) | `@codex review` on GitHub and GitLab | [Bugbot](https://cursor.com/docs/bugbot) | | Security review | `/security-review`, Claude Security plugin | Codex Security, `@codex security review` | [Security Agents](https://cursor.com/docs/security-agents) | | Undo or rewind | Checkpoints (`/rewind`) | Not documented (2026-08-28) | `/rewind` in the CLI | Ultrareview goes furthest: "every reported finding is independently reproduced and verified". A review typically takes 15 to 25 minutes and, after the free runs, costs $5 to $25 in usage credits (Claude Code docs, checked 2026-10-16), so it belongs on high-risk changes, not on every factory pull request. ### Release: CI, triggers and the round trip | Capability | Claude Code | Codex | Cursor | |---|---|---|---| | Run in GitHub CI | `anthropics/claude-code-action@v1` | `openai/codex-action@v1` | Cursor CLI in GitHub Actions | | Run in GitLab | GitLab CI/CD | Merge request review (`@codex review`) | GitLab integration | | Trigger from your own systems | Routines API trigger | Not documented (2026-08-28) | Webhook triggers | | Outbound notification of run status | Not documented (2026-09-26) | Not documented (2026-08-28) | Webhooks (`statusChange`) | | Route and approve pull requests | Not documented (2026-09-26) | Not documented (2026-08-28) | [PR Routing & Approval](/en/cursor-ide/ship/rollouts-and-approval/) ([vendor docs](https://cursor.com/docs/approval-agents)) | | Steer a run from a phone | Remote Control | Codex Remote (`codex remote-control`, experimental) | Cloud Agents on mobile | The gap that matters is the round trip. A routine's API trigger "returns a session URL", but Claude Code documents no outbound webhook, so your systems poll. Cursor documents both directions. ## Where a software factory rots first **The queue outruns review capacity.** The symptom is review wait climbing week on week while throughput looks healthy, which is the Faros pattern. To recover, cap the number of open factory pull requests per loop, pause intake when the cap is reached, and move reviewers to the evidence bundle instead of the diff. **The oracle drifts into the agent's reach.** The symptom is a green run whose diff touches a test, a snapshot or a lint rule. To recover, revert the change, add the path to `CODEOWNERS`, and add a CI step that fails when a factory branch modifies protected paths. **Unattended runs inherit yesterday's permissions.** Routines and automations run without approval prompts by design, so whatever the harness allowed last month is what runs at 3 a.m. To recover, keep the allowlist in version control and review it like code. Set the permission mode in user or managed settings, because Claude Code ignores `auto` and `bypassPermissions` in project settings, and pin Codex's limits in `requirements.toml`. **Untrusted text reaches an agent with tools.** Issue bodies, pull request comments, webhook payloads and scraped pages are written by outsiders. To recover, give loops that read them read-only tools, run their untrusted-input steps with `--bare --setting-sources "" --strict-mcp-config` so no project hooks or MCP servers load (`--bare` authenticates only with `ANTHROPIC_API_KEY` or an `apiKeyHelper`, not an OAuth token), and hand the write step to a separate job. Claude Code already labels routine fire text as untrusted data. **Nobody owns the whole line.** Six stations with six owners and no one reading the seams is how a green pipeline ships a wrong product. To recover, name one owner per loop who runs the weekly sampling audit and can switch the loop back to supervised without a meeting. ## Where to go next at Level 5 - [Level 4: You Write the Specs](/en/ladder/level-4-write-specs/) — The prerequisite: specs, stop conditions and hooks for one long run. - [One map](/en/ladder/one-map/) — Ladder, lifecycle and factory stations in one grid. - [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) — What a human reads when the factory writes the diff. - [Software factories](/en/method/principles/software-factories/) — The full build guide and autonomy ledger. - [From issue to pull request](/en/workflows/automate/issue-to-pr-pipeline/) — A worked end-to-end factory loop in all three tools. - [Shaping a backlog for agents](/en/teams/agent-ready-backlog/) — Slice work into loops an agent can finish and prove. - [Unattended agent runs](/en/cto-scorecard-guide/tier3-overnight-runs/) — The policy for bounded recurring work. - [The human's job](/en/ladder/the-humans-job/) — What no station can decide for you. --- # One map: the autonomy ladder, the lifecycle and the factory stations URL: https://developertoolkit.ai/en/ladder/one-map/ Description: One grid for the autonomy ladder (L0–L5 per loop), the six lifecycle stages and the six factory stations, with the evidence each cell requires. The one map combines three axes in a single grid. Maturity is the autonomy ladder, Level 0 to Level 5, measured per loop. Process is the six lifecycle stages, from plan to maintain. Capability is the six factory stations, from intent to release. Each cell names the evidence a loop must produce before it may run at that level. Your adoption roadmap says "wave 3", the CTO scorecard says "Level 2", a vendor deck says "around L3", and the compound engineering guide says "stage 2". Then the board asks: what level are we at? Four numbers, four definitions, and none says which work may merge without a human reading it. This page reconciles them: one vocabulary for maturity, process and capability, and a map from every older scheme on this site back to it. ## What the one map settles for each reader - **Developers:** which evidence your loop has to produce before you stop reading its diffs, stage by stage. - **Tech leads:** a loop register you can copy, and a rule for promoting or demoting a loop on evidence rather than on confidence. - **CTOs:** a metric definition for team and organization level that cannot be averaged into a vanity number, and risk classes that sit beside levels instead of inside them. - **Executives:** one sentence to report upward ("share of merged change produced at Level 4 or above, with its failure rate") and the list of outside models that are not the same thing. ## What are the three axes of the one map? Each axis answers a different question with a different unit, and confusing the units starts most maturity arguments. | Axis | Question it answers | Values | Unit it applies to | Canonical page | |---|---|---|---|---| | **Maturity** | Who reads what before a change merges? | L0 By hand · L1 Assisted · L2 Paired · L3 Review manager · L4 Spec manager · L5 Dark factory | One **loop** | [The autonomy ladder](/en/ladder/) | | **Process** | Where in the life of a change are we? | Plan · design · build · test · deploy · maintain | One **stage** of the lifecycle | [The AI-native lifecycle](/en/method/lifecycle/) | | **Capability** | What machinery exists to run that stage without a person? | Intent · harness · loop · graph · verification · release | One **station** of the factory | [Level 5: the six stations](/en/ladder/level-5-run-the-factory/) | A **loop** is a repeatable class of change with its own trigger, oracle and stop condition: dependency bumps, flaky-test repair, feature work in the checkout service, schema migrations. A level belongs to a loop, not to a company: a dependency-bump loop with a deterministic check can run at Level 4 while the feature work beside it sits at Level 2, and both answers are correct at once. The ladder's six levels follow Dan Shapiro, "The Five Levels: from Spicy Autocomplete to the Dark Factory" (January 2026); the short names are this site's. Shapiro places most AI-native developers at Level 2 and says almost everyone tops out at Level 3, which is why most cells in the grid below sit in those two columns for most teams. ### Where do the nine task nodes fit? The tool trees (Claude Code, Codex, Cursor) are organized by nine task nodes. They are navigation, not a fourth axis, and each one maps onto the process and capability axes: | Task node in the tool trees | Lifecycle stage | Stations it mostly documents | |---|---|---| | Quick start | Setup, before the lifecycle | Intent, harness | | Plan | Plan, design | Intent | | Build | Build | Harness, loop, graph | | Test | Test | Verification | | Ship | Deploy | Verification, release | | Operate | Maintain | Loop, release | | Automate | Build to maintain | Loop, graph, release | | Team | Across stages (shared rules and people) | Intent | | Reference | None | None | ## The grid: what evidence does each level require at each stage? Rows are lifecycle stages; each row names the stations that produce its evidence. Columns are ladder levels. A cell is the evidence that must exist, and be produced by a machine wherever the level is L4 or above, before a loop may claim that level at that stage. L0 and L1 are left out: at those levels the evidence is the human who typed or accepted each line. | Stage (stations) | L2 Paired | L3 Review manager | L4 Spec manager | L5 Dark factory | |---|---|---|---|---| | **Plan** (intent) | Intent lives in the prompt and the developer's head | A written task with acceptance criteria in the issue | A committed `intent.md` or spec with machine-checkable acceptance criteria, accepted by a named owner before the run | Intents arrive from a queue (tracker, alert, incident), each carrying its own oracle; the owner signs the intent, not the code | | **Design** (intent, graph) | Decided live in the session | A plan reviewed in plan mode before implementation starts | Committed `spec.md` and `plan.md`; architecture constraints encoded as rules or fitness functions the agent cannot edit | Decomposition runs as a versioned workflow; a design change outside the agreed envelope escalates to a human | | **Build** (harness, loop, graph) | The human watches every edit and answers permission prompts | The agent runs unattended in a worktree or sandbox; the evidence is the diff plus a local test run | An explicit permission mode and sandbox for the run type, a stop condition a machine evaluates, and a retry bound | Event-triggered runs with no approval prompts; the harness configuration is versioned and reviewed like code | | **Test** (verification) | Tests the human writes or reads | CI is green and a human reads the diff, tests included | An oracle outside the agent's reach (protected tests, acceptance tests from the spec) and an evidence bundle attached to the pull request | Oracle strength is measured, not assumed; CI rounds are bounded; the factory itself has evals | | **Deploy** (verification, release) | The human merges what they wrote with the agent | A human approves the merge after reading every diff | The merge decision rests on the evidence bundle and the risk class; code is read only for the classes that require it | Low-risk classes auto-merge by rule; progressive delivery with automatic rollback on a service-level breach | | **Maintain** (loop, release) | Humans watch production | The agent triages on request; a human drives the fix | Scheduled or event-triggered runs open pull requests from alerts, with runtime signals in the evidence | A breached control band writes the next intent, and every incident adds a case to the oracle | Read the grid by row, not by column. A loop is at the lowest level any of its stages supports. A loop with an L4 test row and an L3 deploy row is an L3 loop, because a human still reads every diff before merge. The canonical pages for the cells are [evidence instead of diffs](/en/ladder/evidence-not-diffs/) (what a human reads at L4), [the evidence bundle](/en/workflows/ship/evidence-bundle/) (what the pull request must carry), [oracle strength](/en/workflows/test/oracle-strength/) (how strong the test row is) and [progressive delivery](/en/workflows/ship/progressive-delivery/) (the L5 deploy row). ## How do you score a team or an organization? A team does not have a level. It has a **distribution**: the share of its merged change produced by loops at each level. Use this definition as it stands. | Field | Definition | |---|---| | Unit | One merged pull request (or merged change set), tagged with the loop that produced it | | Level of a change | The level the loop held on the merge date, established by the evidence test in the next section, not by self-report | | Window | A rolling 30 days for teams; a quarter for the organization | | Team level | Share of merged changes at L0–L2, L3, L4 and L5 in the window | | Organization level | The same distribution across all teams, plus the number of loops at L4 or above | | Paired stability measure | Change failure rate (or revert rate) per level, in the same window | | Forbidden | Averaging levels into one number ("Level 3.4"), or reporting the best loop as the team's level | The paired stability measure is not optional. DORA's 2025 report (Nathen Harvey and Derek DeBellis, Google Cloud, 23 September 2025) found "a positive relationship between AI adoption on both software delivery throughput and product performance", and in the same breath: "However, AI adoption does continue to have a negative relationship with software delivery stability." A distribution that moves right while failures rise is not progress. An illustrative team, not a measured one: | Loop | Level | Merged PRs, 30 days | Share | |---|---|---|---| | Dependency bumps | L4 | 60 | 30% | | Flaky-test repair | L4 | 20 | 10% | | Checkout feature work | L3 | 84 | 42% | | Schema migrations (critical class, always read) | L3 | 12 | 6% | | Payment-provider integration | L2 | 24 | 12% | The report line is: "Checkout: 40% of merged change at L4, 48% at L3, 12% at L2, with change failure rate per level beside it." The team is not "Level 3". It runs two L4 loops, and the next promotion decision is about one named loop, not the team. ### Run the distribution from your merged pull requests Export the last 30 days of merged pull requests with the GitHub CLI, in your terminal, from the repository root: ```bash # GNU date (Linux); on macOS use: date -v-30d +%F SINCE=$(date -d '30 days ago' +%F) gh pr list --state merged --search "merged:>=$SINCE" --limit 1000 \ --json number,title,labels,author,headRefName,files,commits,reviews,mergedBy,mergedAt,body > merged-prs.json # Slim the export: the raw file is one JSON line, often several MB, which a # file-reading tool cannot page through. jq '[.[] | {number,title,labels:[.labels[].name],author:.author.login,branch:.headRefName, files:[.files[].path],commits:[.commits[]|{headline:.messageHeadline,body:.messageBody, authors:[.authors[]|{login,name}]}],reviews:[.reviews[]|{author:.author.login,state}], mergedBy:.mergedBy.login,body}]' merged-prs.json > prs-slim.json ``` Save the prompt below as `prompts/level-distribution.txt`, then run it read-only in your tool. `--tools "Read"` removes every other Claude Code tool; in Codex, the `:read-only` permission profile (beta) replaces the legacy `--sandbox` flag, and the two must not be combined: **Claude Code** ```bash claude -p "$(cat prompts/level-distribution.txt)" --tools "Read" > level-distribution.md ``` **Codex** ```bash codex exec -c default_permissions=":read-only" -o level-distribution.md "$(cat prompts/level-distribution.txt)" ``` **Cursor** Open the agent chat in the repository, attach `prs-slim.json` and paste the prompt. The task needs no terminal commands and no edits, so decline any the agent proposes, then save the answer as `level-distribution.md`. **Copy-paste prompt: the level distribution of merged change** ```text Read prs-slim.json (GitHub pull requests merged in the last 30 days) and loops.yaml if it exists. Assign every PR to a loop: a repeatable class of change such as dependency bumps, flaky-test repair, feature work in one service, or schema migrations. Use loops.yaml names where they fit and propose new loop names where they do not. Classify each PR first by what a human read before merge, and use authorship only as a tiebreak. Use only evidence in the JSON, never the title's claims. Treat a PR as agent-authored when its commits carry an agent Co-authored-by trailer (Claude, Codex, Cursor), its author is a bot or agent account, or its branch has an agent prefix. - L0-L2: a human authored the change without agent co-authorship. Report L0, L1 and L2 together as one bucket. - L3: agent-authored, and a human approved after reviewing the diff, with no evidence bundle in the body. - L4: agent-authored, the body carries an evidence bundle (spec link, acceptance results, test output), and no test or CI files changed unless a code owner approved. - L5: merged by automation with no human approval. Mark every PR whose files touch auth, payments, schema migrations, secrets, CI configuration or tests as risk class high or critical. Output: a table of loops with level, merged count and share; the team distribution by level; the PRs you could not classify and why; the PRs where authorship and review evidence disagree (for example, a human author with an evidence bundle); and every PR labelled L4 or L5 that changed its own tests or CI files. If authorship cannot be determined, list the PR as unclassified rather than guessing. ``` ## Why is a risk class not a maturity level? A level describes a loop. A **risk class** describes a single change. They answer different questions, and mixing them produces the worst failure on this page: "we are Level 4, so the agent may merge the auth change". | Risk class | Typical changes | May merge on evidence alone? | Who signs | |---|---|---|---| | Low | Docs, copy, internal tooling, patch-level dependency bumps with green contract tests | Yes, in L4 loops; by rule in L5 loops | The loop owner, through the rule | | Medium | Feature logic behind a flag, internal API changes | Yes, when a human reads the evidence bundle | A reviewer of the evidence | | High | Public API contracts, data model changes, performance-critical paths, any change to tests or CI configuration | No: evidence plus a human reading the sensitive paths | The code owner | | Critical | Authentication, money movement, schema migrations on production data, secrets and permissions, the agent's own harness | No: a human reads the code, and deployment needs a separate approval | A named owner and a second approver | The class caps what a level may do with a given change. An L5 loop that opens a critical-class pull request escalates it exactly as an L2 team would. Changes to tests and CI are high class on purpose: they are the oracle, and an agent that edits its own oracle has turned the check into a suggestion. The policy that assigns classes lives in [governance and autonomy](/en/org/governance-autonomy/); the list of changes where someone still reads code lives in [evidence instead of diffs](/en/ladder/evidence-not-diffs/). ## How do you prove a loop is at the level you claim? A level is a claim about evidence, so it can be checked without reading the code. Run this audit before promoting a loop and once a quarter after. 1. **Sample.** Pick 10 merged pull requests from the loop at random from the last window. 2. **Check the evidence for the claimed level.** For L3, each has a human approval after a diff review. For L4, each carries an evidence bundle, its tests and CI files are untouched (or code-owner approved), and the run recorded a stop condition. For L5, each merged by rule with no human approval, and the rollback path has fired at least once in a drill. 3. **Check the four ledger questions** from [software factories](/en/method/principles/software-factories/): what oracle decides done, can the agent fake it, how long until a wrong answer surfaces, and what the blast radius is. 4. **Compare stability.** The loop's change failure rate or revert rate over the window must be no worse than at its previous level. 5. **Record the verdict** in the loop register with the date and the sample, and have the tech lead sign the promotion. A failed check demotes the loop; it does not start a debate. The tech lead signs promotions, and the CTO owns the risk-class policy. The artifact that makes this repeatable is a loop register, one file per repository: ```yaml # loops.yaml: one entry per repeatable class of change - loop: dependency-bumps owner: platform-team trigger: Renovate pull request opened stages: [build, test, deploy] level: L4 oracle: required CI jobs defined outside the paths the agent may write agent_can_edit_oracle: false time_to_surface_wrong_answer: minutes in CI, hours in canary blast_radius: one service, reversible by revert merge_on_evidence: [low] escalate_to_code_reading: [high, critical] last_audit: 2026-09-26, 10 of 10 PRs carried an evidence bundle next_transition: to: L5 exit_criterion: auto-merge for the low class with canary rollback, 60 days without a loop-caused revert - loop: checkout-feature-work owner: checkout-team trigger: ticket moved to Ready with acceptance criteria stages: [plan, design, build, test, deploy] level: L3 oracle: unit and integration tests in the same repository (agent-editable) agent_can_edit_oracle: true merge_on_evidence: [] next_transition: to: L4 exit_criterion: acceptance tests generated from the spec and protected by CODEOWNERS; evidence bundle required by CI ``` **Copy-paste prompt: draft a loop register from repository history** ```text Draft loops.yaml for this repository. Read the last 200 merged pull requests (use the GitHub CLI if it is available, otherwise git log on the main branch), CODEOWNERS, the CI configuration and any AGENTS.md or CLAUDE.md. Group the changes into 4 to 8 loops, each a repeatable class of change with its own trigger. For every loop fill in: owner, trigger, lifecycle stages it covers (plan, design, build, test, deploy, maintain), the current level on the autonomy ladder (L2 to L5) judged only by what a human read before merge, the oracle that decides done, whether an agent can edit that oracle, time until a wrong answer surfaces, blast radius, and the risk classes (low, medium, high, critical) that may merge on evidence alone. Do not promote any loop above what the evidence shows. For each loop, name the single missing piece of evidence that blocks the next level, as an exit criterion a script could check. Do not edit any file except loops.yaml. ``` ## Which older schemes does the one map replace? This site used to run several numbering schemes side by side. They are retired, and each maps back to one axis of the map. If you meet one in an older bookmark or an internal deck, translate it with this table. | Retired scheme | What it described | What replaces it | Where to read it now | |---|---|---|---| | Waves 0–5 | A repository rollout in fixed waves | Ladder transitions, one loop at a time | [Adoption roadmap](/en/teams/adopt/adoption-roadmap/) | | Weeks 1–8, "Month 2+" | A developer's learning calendar | Track steps with exit criteria | [Developer track](/en/start/developer/) | | Weeks 1–12, "Month 4+", "Phased Enterprise Rollout" | An organization rollout calendar | Stop/go gates stated as ladder transitions per loop | [Transformation roadmap](/en/strategy/transformation-roadmap/) | | Adoption Curve, Phase 0, org-size models | Who adopts first, and in what order | Ladder transitions plus a designed pilot | [Pilot design](/en/org/pilot-design/) | | Phases 1–4 | A tool migration checklist | Unnumbered migration steps | [Migration guide](/en/comparison/migration-guide/) | | "Tier 2 / Tier 3" | Parallel agents, then overnight runs | L3 (parallel agents, a human reads every diff) and L4–L5 (unattended runs) | [Team parallelism](/en/cto-scorecard-guide/team-parallelism-tier2/) | | "Reactive → Strategic Leader" | Leadership maturity labels | The Level 1–4 bands the CTO scorecard scores: Assisted, Paired, Review manager, Spec manager | [CTO scorecard guide](/en/cto-scorecard-guide/) | | Every's stage ladder (stages 0 to 5), used as a second ladder | An individual's adoption path | The ladder, per loop; Every's stages remain Every's own model (see below) | [Compound engineering](/en/ecosystem/frameworks/compound-engineering/) | | "Risk tiers 0–3" | How dangerous a change is | Risk classes: low, medium, high, critical | [Governance and autonomy](/en/org/governance-autonomy/) | The CTO scorecard band is a self-assessment of practices (tooling, review, governance), not the level distribution of merged change. Never report it as "our level"; quote it separately from the distribution. The pattern behind the table: a date is not evidence, and a measure of danger is not a measure of maturity. **Other maturity models you'll meet** These models are in circulation and worth knowing. **None of them is the autonomy ladder, and this site does not map them onto it.** Use each for what it measures. - **JetBrains AIDEs Framework** (JetBrains Research blog, September 2026; secondary source: summarised from a search extract, not the original post). Five delegation levels, from L1 "Tool" to L5 "Competence Center", across 35 SDLC activities. An AIDEs "L3" is not ladder Level 3, so never convert a claim such as "tool X is around L3" from one model to the other. - **Every's stage ladder** (Every Inc.'s compound engineering guide at every.to; Kieran Klaassen introduced the method in August 2025). Stages 0 to 5 that describe how one person's working style changes, from writing every line to directing parallel cloud agents. The [compound engineering page](/en/ecosystem/frameworks/compound-engineering/) covers the stages in Every's own terms. - **DORA AI Capabilities Model** (Kevin M. Storer and Derek DeBellis, Google Cloud, 23 September 2025). Seven organizational capabilities, no levels; see [the DORA AI capabilities self-assessment](/en/org/dora-ai-capabilities/). - **DX AI Measurement Framework** (Abi Noda and Laura Tacho, DX; read second-hand through a search extract). Three dimensions, utilization, impact and cost. It is a metrics framework, not a maturity model; its home is [metrics frameworks](/en/org/metrics-frameworks/). **Copy-paste prompt: translate an old roadmap into ladder transitions** ```text Read the attached rollout plan. It uses a calendar or tier scheme (waves, weeks, phases, tiers or maturity labels). Rewrite it as ladder transitions: for each step, name the loop (a repeatable class of change), its current level on the autonomy ladder (L0 to L5), the target level, and the exit criterion as evidence a script could check (for example: "10 of 10 sampled PRs carry an evidence bundle; tests and CI files untouched"). Replace every risk tier with a risk class (low, medium, high, critical) attached to kinds of change, not to teams. Drop every date that is not tied to an exit criterion. List anything in the original plan that measured activity (seats, prompts, tokens) rather than evidence, and say what to measure instead. ``` ## Where the one map goes wrong **Averaging levels into one number.** "The org is at Level 3.2" hides the two loops that could run dark and the critical loop that should never have left L3. Recovery: report the distribution and the count of loops at L4 or above, never a mean. **Scoring the team by its showcase loop.** One polished dependency-bump loop at L4 becomes "we are an L4 team". Recovery: the team level is the share of merged change, so a loop that produces 5% of merges moves the distribution by 5%. **Reading a risk class as permission.** A loop promoted to L4 starts merging schema migrations on evidence alone. Recovery: classes cap levels; add the critical paths to CODEOWNERS and make CI fail when an evidence-only merge touches them. **Claiming a level the stations cannot support.** A loop is called L4 while its oracle sits in the same diff the agent writes. Recovery: run the audit above; a loop whose agent can edit its oracle is L3 until the oracle moves out of reach. **Moving right while getting worse.** The share at L4 rises and so does the revert rate. Recovery: demote the loop that caused the reverts, fix its oracle, and re-audit before promoting it again. ## Where to go next from the one map - [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) — What a human reads at Level 4, and the change classes where someone still reads the code. - [Level 3: You Review the Diffs](/en/ladder/level-3-review-diffs/) — Where most loops sit today, and why review bandwidth is the ceiling. - [Level 5: the six stations](/en/ladder/level-5-run-the-factory/) — Each factory station mapped to what Claude Code, Codex and Cursor document. - [The AI-native lifecycle](/en/method/lifecycle/) — The six stages and the artifact each one commits. - [DORA, SPACE, DX Core 4 and AI measurement](/en/org/metrics-frameworks/) — Canonical definitions for the stability measure you report beside the distribution. - [The economics of agent-built software](/en/strategy/economics/) — Cost per accepted change, and what the distribution means for it. --- # The State of Agentic Engineering, September 2026 URL: https://developertoolkit.ai/en/ladder/state-of-agentic-engineering-2026/ Description: How much code agents write in 2026, where the autonomy ladder tops out, and why verification is the limit. Third-party evidence only, every number dated. Agentic engineering is the discipline of directing fallible coding agents without lowering the quality bar. As of September 2026, DX measures 51.9% of code as AI-authored across 400+ companies and Sonar's developers report 42%. Anthropic reports over 80% of its merged lines, and Faros and DORA point to verification capacity, not generation, as the limit. This report is for CTOs and executives setting strategy, tech leads deciding how far to trust agents, and developers placing their own workflow. A CTO asks what share of the codebase agents write. The lead says 40%. The developer running the agents knows half of those lines are tests the agent wrote for its own change. Three people, three rungs of one ladder, and nobody can name the rung. ## What you'll walk away with from the 2026 evidence - An evidence table where every number has a publisher, a date and a unit - Shapiro's six-rung autonomy ladder and a five-question test that places a repository on it - The six stations of a software factory, with the matching feature in Claude Code, Codex and Cursor - METR, Faros, DORA and SlopCodeBench, read correctly - Three copy-paste prompts and a metric-pair template for leadership ## How much code do agents write in 2026? The share rises everywhere it is measured. | Measure | Then | Now | Source | |---|---|---|---| | Anthropic merged lines authored by Claude | "low single digits", Feb 2025 | "more than 80%" | Anthropic, "as of May 2026" | | AI-authored share, 400+ companies | 27.4%, Q1 2026 | 51.9% average | DX, 17 Jun 2026 | | Committed code reported as AI-generated or assisted | not measured | 42% | Sonar, 8 Jan 2026 | | Stripe PRs per week with no human-written code | "over a thousand", 9 Feb 2026 | "over 1,300" | Stripe, 19 Feb 2026 | | Code merged per Anthropic engineer per day | 2024 baseline | 8× | Anthropic, Q2 2026 | | Merged PRs of CLI-agent adopters at Microsoft | same engineers without the tools (modelled) | "roughly 24% more" | Murphy-Hill et al., arXiv, 1 Jul 2026 | | Merged PRs and share reverted, Claude Code's own codebase | June 2026 | PRs "more than doubled"; share reverted fell "by more than 40 percent", Sep 2026 | Anthropic, "How the Claude Code team does code review", 15 Oct 2026 | | Code changes merged per day, Anthropic's monorepo (all changes; no agent share given) | "around 2,500", May 2026 | "over 30,000", Sep 2026 | Anthropic, 13 Oct 2026 (cited by the 15 Oct article) | At the individual end the share reaches 100% and stays there. Boris Cherny, who leads Claude Code, told Fortune on 29 January 2026: "For me personally, it has been 100% for two+ months now, I don't even make small edits by hand." Dario Amodei told the Council on Foreign Relations on 10 March 2025 that AI would write "essentially all of the code" within twelve months. Eighteen months on, Anthropic reports "more than 80%" of its own merged lines, and the industry's best series is DX's 51.9%. Fast Company reports Sundar Pichai putting Google's new code at 75% in April 2026, a secondary report that no Google primary source confirms. ### What do the percentage-of-code numbers actually measure? Every one of those percentages counts lines, and lines are the cheapest thing an agent makes. Ryan Greenblatt of Redwood Research wrote on 22 October 2025: "The productivity boost at a given fraction of code generated isn't that high because AI allows people to cheaply generate lots of very low value code." Anthropic draws the same line. Its leadership has estimated 90% or more "including scripts and experimental code", while the published figure measures "the share of lines merged to production that can be attributed to Claude." DX adds the cost: median pull request size grew "from 44 lines to 72 lines per pull request between July 2025 and June 2026". A rising AI share and a near-doubled diff are one event, and the reviewer pays for it. ## What is agentic engineering? The phrase was in use before Andrej Karpathy endorsed it. Mario Zechner wrote of "what people call 'agentic engineering'" on 2 June 2025, and Peter Steinberger titled a post with it on 14 October 2025. Karpathy called it "personally my current favorite" on 4 February 2026. In his published transcript of his Sequoia Ascent talk (30 April 2026; the page says an LLM cleaned up the transcript and that he read it) he explains the choice: > "I call it agentic engineering because it is an engineering discipline. You have agents, which are spiky entities. They are fallible and stochastic, but extremely powerful. How do you coordinate them to go faster without sacrificing your quality bar? > > Vibe coding raises the floor. Agentic engineering is about extrapolating the ceiling." The constraint is explicit: "You are still responsible for your software, just as before." What changed is the unit of work. His published summary, which the page says an LLM wrote, puts it as "from typing lines of code to delegating larger 'macro actions'." The definition at the top of this page condenses that passage; it is not a quotation. Delegating whole tasks to an agent that plans, writes, runs and iterates is [agentic coding](/en/glossary/agentic-coding/); that name describes what the tool does, and agentic engineering adds the responsibility Karpathy names. The [vibe coding vs agentic engineering comparison](/en/ladder/vibe-coding-vs-agentic-engineering/) sets agentic engineering against the 20 other names in use. ## Which level of the autonomy ladder is your team at? Dan Shapiro published the model on 23 January 2026 as "The Five Levels: from Spicy Autocomplete to the Dark Factory", after the NHTSA driving-automation scale. It is zero-indexed, so six rungs sit under a title that says five. Simon Willison's write-up of 28 January identifies the Level 5 team as StrongDM's AI division. | Level | Shapiro's name | Writes | Reads | What it feels like | Guide | |---|---|---|---|---|---| | L0 | Spicy autocomplete | Human | Human | "not a character hits the disk without your approval" | [L1–2](/en/ladder/level-1-2-assisted-paired/) | | L1 | The coding intern | Human | Human | "you offload specific, discrete tasks to your AI intern" | [L1–2](/en/ladder/level-1-2-assisted-paired/) | | L2 | The junior developer | Both | Human | "feels like you are done. But you are not done" | [L1–2](/en/ladder/level-1-2-assisted-paired/) | | L3 | The developer | AI | Reviewer | "Your life is diffs." | [L3](/en/ladder/level-3-review-diffs/) | | L4 | The engineering team | AI | Tests | "leave for 12 hours, and check to see if the tests pass" | [L4](/en/ladder/level-4-write-specs/) | | L5 | The dark software factory | AI | Nobody | "It's a black box that turns specs into software." | [L5](/en/ladder/level-5-run-the-factory/) | Shapiro calls Level 2 "where 90% of 'AI-native' developers are living right now", says "almost everyone tops out here" of Level 3, places himself at Level 4, and puts "a handful of people" at Level 5, where "humans are neither needed nor welcome." Answer each question with evidence, not impressions. The first one you cannot answer marks the ceiling. 1. **L0 to L1.** Does code reach disk that nobody typed? 2. **L1 to L2.** Do you hand over whole tasks with acceptance criteria, or only completions inside a function you are writing? 3. **L2 to L3.** Does the agent run unattended long enough that you meet its work as a diff? That is a job change, not a speed change. 4. **L3 to L4.** Is there a stop condition a machine can evaluate, written as commands that exit 0? Level 4 is defined by leaving. 5. **L4 to L5.** Does anything merge that no human read, and can you name the oracle that made it safe? Assign a level per loop, not per company. A dependency-upgrade loop with a deterministic validator can run at Level 4 while feature work in the same repository stays at Level 2. The [one-map page](/en/ladder/one-map/) lays the ladder over the delivery lifecycle. ## What does a production software factory look like? Stripe's is the most detailed public account. Its Minions are one-shot, end-to-end agents built on "a fork of Block's coding agent goose", drawing on a Toolshed of "nearly 500 MCP tools". By 19 February 2026, "over 1,300 Stripe pull requests... merged each week are completely minion-produced, human-reviewed, but containing no human-written code." Three design decisions are worth copying: - **The loop has a hard bound.** Stripe allows "at most two rounds of CI", then "we send the branch back to its human operator for manual scrutiny." - **The workflow is code.** Blueprints are "a state machine that intermixes deterministic code nodes and free-flowing agent nodes." - **The oracle predates the agents.** The minions run against "Stripe's enormous preexisting battery of tests — over three million of them." Microsoft's study of its early-2026 rollout of Claude Code and GitHub Copilot CLI covers "tens of thousands of engineers", and its authors qualify the 24% themselves: "a merged PR is not the same as the value it delivers." Factories cost real money: an hour of Steve Yegge's Gas Town orchestrator cost DoltHub's Tim Sehn "about $100 in Claude tokens" (15 January 2026), and a week cost $3,000. BCG Platinion wrote in March 2026 that "the defining shift is not the absence of humans; it is the relocation of human effort." ### The six stations of a software factory Alexey Grigorev's July 2026 taxonomy separates context engineering (what the agent knows before it starts), loop engineering (when it stops) and graph engineering (who does what once there is more than one agent). Add what may run, what proves the result and what ships, and a factory has six stations, each with a documented feature in every tool. | Station | Decides | Claude Code | Codex | Cursor | |---|---|---|---|---| | Intent | What it knows first | [`CLAUDE.md`](https://code.claude.com/docs/en/memory.md) | [`AGENTS.md`](https://learn.chatgpt.com/docs/agent-configuration/agents-md.md) | [Rules](https://cursor.com/docs/rules) | | Harness | What runs per edit | [Hooks](https://code.claude.com/docs/en/hooks-guide.md) | [Hooks](https://learn.chatgpt.com/docs/hooks.md) | [Hooks](https://cursor.com/docs/hooks) | | Loop | When it stops | [`/goal`](https://code.claude.com/docs/en/goal.md) | [`/goal`](https://learn.chatgpt.com/docs/developer-commands.md) | [`/goal`](https://cursor.com/docs/cli/reference/slash-commands) | | Graph | Who does what | [Subagents](https://code.claude.com/docs/en/sub-agents.md) | [Subagents](https://learn.chatgpt.com/docs/agent-configuration/subagents.md) | [Subagents](https://cursor.com/docs/subagents) | | Verification | What proves it | [`/code-review`](https://code.claude.com/docs/en/code-review.md) | [`/review`](https://learn.chatgpt.com/docs/code-review.md) | [Bugbot](https://cursor.com/docs/bugbot) | | Release | How it ships | [GitHub Actions](https://code.claude.com/docs/en/github-actions.md) | [`openai/codex-action@v1`](https://github.com/openai/codex-action) | [GitHub Actions](https://cursor.com/docs/cli/github-actions) | Claude Code cells were re-checked on 26 September 2026. Codex features were confirmed the same day with `codex-cli 0.157.1` (`codex features list` shows `goals`, `hooks` and `multi_agent` stable and on); Codex documentation links and Cursor cells were last verified on 28 August 2026. In Claude Code, spawning a subagent fails once 20 are running in a session (v2.1.217+; `CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS` changes the limit, and ultracode sessions are exempt). Cursor began rolling out `/goal` on 19 August 2026 (per its changelog, read 28 August 2026; Cursor's docs still call it rolling out, checked 2026-10-12), the same day subagents gained their own virtual machines. `/loop` is built into Claude Code and has been a bundled skill in Cursor since 3.5 (20 May 2026); Codex 0.162.1 has no `/loop` (checked 2026-10-12) and covers scheduled cadence with Automations (the goal-directed loop is `/goal`). ## What does the evidence against autonomy say? Four bodies of evidence argue for caution; quote each one whole. **METR measured a slowdown, then lost its instrument.** The 2025 randomised trial of 16 experienced open-source developers across 246 issues found tasks took 19% longer (interval +2% to +39%), while the developers believed afterwards they had been sped up by 20%. The February 2026 follow-up reports changes in completion time of -18% (interval -38% to +9%) for returning developers and -4% (-15% to +9%) for new recruits. Negative means less time, so both point estimates favour AI, but both intervals cross zero and METR says the new data "gives us an unreliable signal of the current productivity effect of AI tools." Neither "18% slower" nor "18% faster" is a finding. **Faros measured the whiplash.** Two years of telemetry from 22,000 developers and 4,000+ teams, published April 2026. Throughput rose: epics per developer +66.2%, tasks +33.7%, merge rate +16.2%, while deployments per week fell 11.7%. Quality fell in the same window: bugs per developer +54%, incidents per pull request +242.7%, churn +861%, pull requests merging unreviewed +31.3%. Median time in review rose 441.5% and time to first review 156.6%; those are two different metrics. **DORA named the mechanism.** Its 2025 report found "a positive relationship between AI adoption on both software delivery throughput and product performance" and that "AI adoption does continue to have a negative relationship with software delivery stability." The reason: "Without robust control systems, like strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability." **SlopCodeBench measured decay over time.** Across 36 problems, 196 checkpoints and 15 agents extending their own earlier solutions, "no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints". ### How is agent output verified without reading every line? None of that argues for staying at Level 2. It shows the climb: autonomy is earned per loop against a verification oracle, never declared for a codebase. Faros shows generation scaling while verification did not, which is the Level 3 ceiling in telemetry. Climbing means adding a proof, not removing a reader. | Layer | What it proves | Who signs off | |---|---|---| | Stop condition as commands that exit 0 | The loop's stated goal holds | The spec author, before the run starts | | Deterministic validators (types, lint, tests, schema checks) | Known classes of error are absent | CI, on every push | | Review agent (`/code-review`, `/review`, Bugbot) | A second model found no blocking issue | The human assignee reads its findings, not the diff | | Retry bound and escalation | The agent cannot loop forever or weaken a test to pass | The human operator, after the bound (Stripe: two CI rounds) | | Production signals (incidents per PR, rollback rate) | The oracle is catching what matters | The tech lead, weekly | Naresh B A puts the rule in one line: "A model can argue that its work is complete. A deterministic validator can prove that a required field is missing." Linear's product rule: "issues can only be assigned to humans, and only delegated to agents." The full method is [reading evidence instead of diffs](/en/ladder/evidence-not-diffs/). Anthropic's Claude Code team describes a similar stack of layers from inside the Claude Code repositories (CLI, desktop and web apps, mobile apps). Its engineering article of 15 October 2026 says Claude writes "about 90 percent of Claude Code's own code" (the article does not say what is counted, so read the figure with the lines-are-not-work caveat above) and that from June to September 2026 merged pull requests "more than doubled while the share that had to be reverted fell by more than 40 percent". Both are a vendor reporting on itself, and the article gives no absolute counts. It also names its own gap: no circuit breakers yet that tighten approval when revert or incident rates rise, and its engineers want them before more code moves to zero-human merges. The approval tiers behind it are on [governance and autonomy](/en/org/governance-autonomy/). ## What stays human in agentic engineering? Six responsibilities survive every rung below the dark factory: taste, architecture, product direction, stop-condition design, permissions and the decision not to automate. In his published transcript, Karpathy says "you are in charge of taste, engineering, design, and whether the system makes sense." Understanding is the one that erodes quietly. In Anthropic's randomised trial of 52 mostly junior engineers learning a new library (29 January 2026), "the AI group averaged 50% on the quiz, compared to 67% in the hand-coding group." Oversight needs the skill that over-delegation wears down. [The human's job](/en/ladder/the-humans-job/) covers each responsibility in depth. ## Copy-paste prompts and templates for placing your team on the ladder Run these prompts against a repository you ship from; each works unchanged in Claude Code, Codex and Cursor. **Copy-paste prompt: audit this repository's autonomy level** ```text Read this repository's CI configuration, agent instruction and settings files (CLAUDE.md, AGENTS.md, .claude/settings.json, any .codex/ config, Cursor rules), test scripts and the last 30 merged pull requests. Levels: L0 AI only autocompletes; L1 human writes, AI does discrete tasks; L2 human and AI pair, human reads every line; L3 AI writes, human reviews diffs; L4 AI writes, tests decide, human writes specs; L5 nobody reads the code. For each level from L1 to L5, list the concrete evidence for and against this repository operating there, with file paths. Name the single verification gap that stops it from moving one level up, and write the smallest change that closes it: a hook, a test target or a stop condition. Do not edit any files. ``` **Copy-paste prompt: design the stop condition for a /goal run** ```text I am about to start a goal-directed run to migrate every call site of our deprecated HTTP client to the new one. Write the stop condition as shell commands that must all exit 0 (tests, typecheck, lint, and a grep that finds no remaining import of the old client). List the files and directories the agent may not touch. Define a one-line checkpoint log per round, and a budget of rounds after which the run stops and reports instead of continuing. Reject any condition the agent could satisfy by deleting or weakening a test. ``` **Copy-paste prompt: write the verification contract for one factory station** ```text Station: verification. Write its contract as a markdown table: the inputs it accepts, the deterministic validators that must pass before hand-off (with the exact commands for this repository), the failure route (which station receives a FAIL and what it gets back), the maximum retries, and the trigger for escalating to a human. Then list the validator commands on their own so I can paste them into CI. ``` For leadership, the adoptable artifact is a reporting rule: never show a share-of-code or throughput number without its quality pair. | Report this | Always next to | Why the pair | |---|---|---| | AI-authored share of merged lines | Incidents per pull request | Anthropic and Redwood: lines are not work | | Pull requests merged per engineer | Median time in review and time to first review | Faros: both review metrics moved with throughput | | Median pull request size | Share of PRs merged without review | DX and Faros: bigger diffs, thinner review | | Autonomy level per loop | The named oracle and its retry bound | Stripe: a two-round CI bound plus 3M+ tests is what lets humans review instead of rewrite | DORA's AI Capabilities Model (23 September 2025) names seven foundations to fund first. [Metrics frameworks](/en/org/metrics-frameworks/) turns the pairs into a dashboard. ## When the 2026 agentic engineering numbers get misused Five common misreadings, each with its fix. - **Reading "percentage of lines" as "percentage of work".** Anthropic's footnote and Redwood's critique both refuse that step. Pair every share figure with a work figure. - **Quoting half of Faros.** The +66.2% epics and the +242.7% incidents come from one dataset. Quote throughput and quality together, or neither. - **Reading METR's sign backwards.** METR reports a change in completion time, so -18% means less time, not a slowdown. Quote the interval and METR's "unreliable signal" caveat every time. - **Citing a vendor's internal share as an industry rate.** Anthropic, Stripe, Google and Cherny all report on themselves. For an industry number, use DX's 51.9% or Sonar's 42%. - **Publishing feature claims that have expired.** Cursor had no `/goal` before it began rolling out on 19 August 2026. Re-check the vendor page, or run the CLI's `--help`, on the day you publish, and date every negative claim. ## Where to go next on the ladder - [One map: ladder, lifecycle and stations](/en/ladder/one-map/) - [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) - [The autonomy ladder](/en/ladder/) - [Level 4: you write the specs](/en/ladder/level-4-write-specs/) - [The economics of agentic engineering](/en/strategy/economics/) - [The business case](/en/strategy/business-case/) - [Software factories](/en/method/principles/software-factories/) - [CTO scorecard](/en/cto-scorecard/) - [Developer scorecard](/en/scorecard/) ## Sources 1. Shapiro, "The Five Levels: from Spicy Autocomplete to the Dark Factory", 23 Jan 2026. https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/ 2. Simon Willison, "The Five Levels: from Spicy Autocomplete to the Dark Factory", 28 Jan 2026. https://simonwillison.net/2026/Jan/28/the-five-levels/ 3. Karpathy, "Sequoia Ascent 2026", 30 Apr 2026; the page says an LLM produced its summary and cleaned-up transcript, which Karpathy read. https://karpathy.bearblog.dev/sequoia-ascent-2026/ 4. Anthropic, "When AI builds itself", undated ("as of May 2026"). https://www.anthropic.com/institute/recursive-self-improvement 5. Fortune, "Top engineers at Anthropic, OpenAI say AI now writes 100% of their code", 29 Jan 2026. https://fortune.com/2026/01/29/100-percent-of-code-at-anthropic-and-openai-is-now-ai-written-boris-cherny-roon/ 6. DX, "AI-authored code has nearly doubled", 17 Jun 2026. https://newsletter.getdx.com/p/ai-authored-code-has-nearly-doubled 7. Sonar, "State of Code Developer Survey", 8 Jan 2026. https://www.sonarsource.com/blog/state-of-code-developer-survey-report-the-current-reality-of-ai-coding/ 8. Murphy-Hill, Butler and Savelieva, "Adoption and Impact of Command-Line AI Coding Agents", arXiv:2607.01418, 1 Jul 2026. https://arxiv.org/abs/2607.01418 9. Stripe, "Minions", parts 1 and 2, 9 and 19 Feb 2026. https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents-part-2 10. Ryan Greenblatt, Redwood Research, "Is 90% of code at Anthropic being written by AIs?", 22 Oct 2025. https://blog.redwoodresearch.org/p/is-90-of-code-at-anthropic-being 11. METR, early-2025 developer study, 10 Jul 2025, and uplift update, 24 Feb 2026, with the analysis code that defines the sign. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/, https://metr.org/blog/2026-02-24-uplift-update/ and https://github.com/METR/Measuring-Late-2025-AI-on-OSS-Devs 12. Faros AI, "The Acceleration Whiplash", Apr 2026. https://www.faros.ai/blog/ai-acceleration-whiplash-takeaways 13. Google Cloud, "Announcing the 2025 DORA Report", 23 Sep 2025, and "Introducing DORA's inaugural AI Capabilities Model", 23 Sep 2025. https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report and https://cloud.google.com/blog/products/ai-machine-learning/introducing-doras-inaugural-ai-capabilities-model 14. Orlanski et al., "SlopCodeBench", 25 Mar 2026. https://arxiv.org/abs/2603.24755 15. Anthropic research (Judy Hanwen Shen and Alex Tamkin), randomised trial on AI assistance and coding skills, 29 Jan 2026. https://www.anthropic.com/research/AI-assistance-coding-skills 16. BCG Platinion, "The Agentic Software Factory", 26 Mar 2026. https://www.bcgplatinion.com/insights/the-agentic-software-factory 17. Grigorev, "AI-Native Development", 22 Jul 2026. https://aishippingblog.com/p/ai-native-development-specifications 18. Naresh B A, "Graph Engineering for AI Coding Agents", 30 Jul 2025. https://dev.to/naresh_007/graph-engineering-for-ai-coding-agents-beyond-prompt-loops-48h4 19. Linear, "Our approach to building the Agent Interaction SDK", 1 Aug 2025. https://linear.app/now/our-approach-to-building-the-agent-interaction-sdk 20. DoltHub (Tim Sehn), "A Day in Gas Town", 15 Jan 2026, and "A Week in Gas Town", 24 Mar 2026. https://www.dolthub.com/blog/2026-01-15-a-day-in-gas-town/ and https://www.dolthub.com/blog/2026-03-24-a-week-in-gas-town/ 21. Council on Foreign Relations, "CEO Speaker Series With Dario Amodei", 10 Mar 2025. https://www.cfr.org/event/ceo-speaker-series-dario-amodei-anthropic 22. Fast Company (secondary), "Google CEO says 75% of the company's code is AI-generated", 24 Apr 2026. https://www.fastcompany.com/91531519/google-ceo-says-75-of-the-companys-code-is-ai-generated 23. Karpathy, X post endorsing "agentic engineering", 4 Feb 2026. https://x.com/karpathy/status/2019137879310836075 24. Mario Zechner, blog post using "agentic engineering", 2 Jun 2025. https://mariozechner.at/posts/2025-06-02-prompts-are-code/ 25. Peter Steinberger, "Just Talk To It - the no-bs Way of Agentic Engineering", 14 Oct 2025. https://steipete.me/posts/just-talk-to-it 26. Addy Osmani, Morgan Westlee Lunt and Qing Wang, "How the Claude Code team does code review", claude.dev Blog (Engineering), 15 Oct 2026; vendor self-report; its monorepo figures come from source 27. https://claude.dev/blog/solving-code-review-at-anthropic/ 27. Anthropic, "Redesigning source control at Anthropic", 13 Oct 2026; vendor self-report. https://www.anthropic.com/engineering/redesigning-source-control --- # The Human's Job in Agentic Engineering URL: https://developertoolkit.ai/en/ladder/the-humans-job/ Description: Six decisions that stay human when agents write the code, how to record who owns each loop, and how to prove those decisions still hold. The human's job in agentic engineering is the set of decisions no verification oracle can make for an agent: taste, architecture, product direction, the design of the evals and stop conditions, the permissions agents run with, and the call on which loops must not run unattended. Accountability for each decision stays with a named person. Your team's agents now open more pull requests in a day than the team used to write in a week. Review has quietly turned into approval, an overnight loop ran with whatever permissions the last session left behind, and nobody can say who decided that. This page is for the developer who wants to know what their job is now, the tech lead who has to write it down for a team, and the CTO or executive who has to answer for it. ## What you get from mapping the human's job - The six responsibilities that do not transfer to an agent, each traced to a named source. - A four-question test that decides, per loop, whether it may run unattended. - An autonomy register row you can paste into your handbook, so every unattended loop has an owner, a reason and a review date. - The per-tool commands that turn a permission decision into configuration. - Four checks that prove the human decisions still hold without anyone reading every diff. - Three copy-paste prompts, and a recovery step for each way the human's job quietly disappears. ## Why does the human's job move instead of shrinking? The work changes shape. Andrej Karpathy's published summary of his Sequoia Ascent talk (30 April 2026) puts it as "The unit of programming changed from typing lines of code to delegating larger 'macro actions'." BCG Platinion, writing as consultants in March 2026, says the same in a sentence a board understands: "The defining shift is not the absence of humans; it is the relocation of human effort." Simon Willison proposed [vibe engineering](/en/glossary/vibe-engineering/) for experienced engineers who work faster with agents while staying accountable for the software; he later said agentic engineering was winning as the name. At Level 2 on the [autonomy ladder](/en/ladder/one-map/), all six responsibilities below hide inside the act of writing code, so they feel automatic and nobody names them. From Level 3 upward, each one has to be written into a file, a gate or a policy. An agent does not infer them, and a passing test suite does not report their absence. ## Which six responsibilities stay human? | Responsibility | The decision | Where it is written down | Source | | --- | --- | --- | --- | | Taste | This working implementation is the wrong one | Architecture brief, review rubric | Karpathy (transcript), April 2026 | | Architecture | Boundaries, layering and dependency direction | Instruction file, fitness functions | Karpathy (transcript), April 2026; DORA, September 2025 | | Product direction | What to build, and what should not exist | Spec, acceptance criteria | Karpathy (transcript), April 2026; Shapiro, January 2026 | | Eval and stop-condition design | What "done" means, as commands that exit 0 | Test suite, eval suite, CI config | Shapiro; Stripe, February 2026; Naresh, July 2025 | | Permissions | What an agent may run without asking | Settings files, sandbox profile | Claude Code and Codex docs | | The call not to automate | Which loops run unattended, and who answers for them | Autonomy register | Linear, August 2025 | ### Taste In his published transcript, Karpathy is direct about the agents' present standing: "Right now the agents are like interns. You still have to be in charge of aesthetics, judgment, taste, and oversight." Taste is the judgement that a working implementation is the wrong one: the abstraction that will cost a year, the dependency that solves today's problem and owns tomorrow's, the API that is correct and unusable. No test fails on any of that, which is why it stays with a person. ### Architecture "You still have to understand the fundamentals," Karpathy says in his talk (published transcript), and "you cannot be a good director if you do not understand." Architecture has the slowest feedback loop in the job: wrong answers surface months after the run that made them. DORA's 2025 report (September 2025) found that "Teams working in loosely coupled architectures with fast feedback loops see gains, while those constrained by tightly coupled systems and slow processes see little or no benefit." The architecture decides how much of the acceleration a team keeps. ### Product direction In his published transcript, Karpathy names the bottleneck: "I am becoming the bottleneck of even knowing what we are trying to build, why it is worth doing, and how to direct my agents." Dan Shapiro's Level 4 human (The Five Levels, January 2026) writes a spec and works more like an engineering or product manager than a coder. An agent builds what the spec says with more patience than any team. It does not tell you the feature should not exist. ### Eval and stop-condition design This responsibility grows as the others shrink. Shapiro's Level 4 is "leave for 12 hours, and check to see if the tests pass", which is only safe if somebody designed the tests. Naresh (DEV Community, July 2025) states the case for determinism: "A model can argue that its work is complete. A deterministic validator can prove that a required field is missing." Stripe's Minions, described by Alistair Gray in February 2026, run against "over three million" existing tests for "at most two rounds of CI", after which "we send the branch back to its human operator for manual scrutiny." The oracle is the one part of the system an agent must never author. Choosing it is a human job; keeping it out of the agent's reach is a design constraint. [Protect the oracle](/en/workflows/test/protect-the-oracle/) shows how. ### Permissions What an agent may run unsupervised is a decision every tool asks you to make. Codex documents an approval policy for "when Codex must ask you before it executes an action (for example, leaving the sandbox, using the network, or running commands outside a trusted set)". Claude Code's auto mode routes "tool calls through a classifier that blocks anything irreversible, destructive, or aimed outside your environment". Cursor documents run modes for the same choice. Defaults are not a policy. From Claude Code v2.1.283 (both release channels on 2026-10-12), interactive sessions on supported models start in auto mode, while `claude -p` starts in Manual (auto mode on third-party providers or with telemetry off, from v2.1.285); Codex's default approval policy is `on-request`. An unattended run at 3 a.m. inherits whichever of those applies, and that is a decision somebody made by not making it. ### The call not to automate Linear's write-up on its Agent Interaction SDK (Leela Senthil Nathan, August 2025) states the principle plainly: "an agent cannot be held accountable." The product enforces it: "issues can only be assigned to humans, and only delegated to agents", so "the issue still has a human assignee—someone accountable for the result—but it also has a delegated agent responsible for taking action." Accountability does not delegate. Somebody decides what runs unattended and answers for it, and that decision has a shape: a named person per loop, a written reason, and a date to revisit it. The loops that hurt teams are rarely the ones somebody chose to automate. They are the ones that became unattended because nobody said otherwise. ## Decide which loops may run unattended The four questions of the autonomy ledger, from [software factories](/en/method/principles/software-factories/), decide it per loop. A loop is a named workflow, such as "dependency upgrades in the payments service", not a whole repository. 1. **Name the oracle that decides "done".** "`npm test` and the contract suite exit 0" is an oracle. "It looks right" is not. 2. **Check whether the agent can edit that oracle.** If the agent can change the tests that judge its work, the oracle is a suggestion. Move the tests under code-owner review or make the decisive check predate the change. 3. **Measure how long a wrong answer takes to surface.** Seconds in CI is a green light. A customer report three weeks later is a red one. 4. **State the blast radius.** Reversible and contained, such as a copy change behind a flag, or irreversible, such as a schema migration. 5. **Record the answers in the autonomy register** with an owner, a reason and a review date (template below). A loop that answers cleanly on all four may run unattended. A loop that fails question 2 or 3 stays supervised, however good the harness is. Five classes of work fail those questions often enough to name up front: - **Schema, auth, pricing and migrations.** The oracle is production behaviour, feedback takes days, and the blast radius is customers. A human reads the diff every time. - **Work with no oracle that runs in CI.** Architectural refactors are the standard case: nothing you can run says the new shape is better. Put the effort into the plan, not the automation. - **Work where the agent owns both sides.** A feature in a well-covered module looks safe until you notice the agent wrote the feature and the tests that pass it. Either a human reads the plan, or the decisive check predates the change. - **Loops with slow feedback.** If a wrong answer surfaces when a person notices rather than when a command exits non-zero, the loop accumulates errors quietly. - **Anything that reads text an outsider can write.** Issue bodies, pull request comments, webhook payloads, scraped pages and inbound email are attacker-controlled input. An unattended agent with tools and a prompt-injection surface is a remote-execution path. If the loop must run, treat the input as data, and give the loop the narrowest permission set in the system; the [agent threat model](/en/org/agent-threat-model/) covers the rest. ## Record the decision in the autonomy register The autonomy register holds one row per loop; [the operating model](/en/org/operating-model/) defines who owns it and [governance and autonomy](/en/org/governance-autonomy/) defines the risk classes. Paste this row format into your handbook and fill one row per unattended loop: ```text AUTONOMY REGISTER ROW — one per loop Loop: dependency upgrades, services/payments Ladder level: 4 (agent opens PR; human reads the evidence, not the diff) Risk class: medium Oracle: unit + contract suites in CI; lockfile audit Oracle owner: @payments-leads (CODEOWNERS on tests/ and contracts/) Agent can edit the oracle: no — tests/ and contracts/ are denied to the agent identity Feedback time: under 15 minutes (CI) Blast radius: one service; revert is a single PR Permission set: workspace-write sandbox, registry network only, no prod secrets Outside input: none (reads the lockfile and advisories only) Accountable: @payments-tech-lead (one named person, not a team) Reason: 120 upgrade PRs a quarter; CI catches the failures we have seen Review by: 2026-12-31 ``` The review date matters most. A row whose date has passed is a loop running on an expired decision, so treat it the way you treat an expired certificate. ## Turn the permission decision into configuration The register says what a loop may do; the tool configuration is where that decision becomes true. These are the unattended CI shapes from [permissions and sandboxing](/en/workflows/harness/permissions-and-sandboxing/), checked against Claude Code 2.1.286 and Codex CLI 0.157.1 on 2026-09-26. **Claude Code** `dontAsk` denies anything that would prompt, so the allowlist adds only what the run needs, and the `Edit(/tests/**)` deny keeps the oracle out of reach: ```bash claude -p "Run the test suite and fix failing tests in src/. Do not edit tests." \ --permission-mode dontAsk \ --permission-prompts none \ --settings '{"sandbox":{"enabled":true,"allowUnsandboxedCommands":false}}' \ --allowedTools "Read" "Edit" "Bash(npm test)" "Bash(npm run lint)" \ --disallowedTools "Edit(/tests/**)" ``` For interactive sessions that run in auto mode, `claude auto-mode config` prints the effective auto mode configuration as JSON, so you can diff the policy you approved against what is running. **Codex** `codex exec` has no `-a` of its own in 0.157.1, so the approval flag goes before the subcommand. The `:workspace` permission profile (beta) grants workspace writes and no network: ```bash codex -a never exec -c 'default_permissions=":workspace"' \ "Run the unit tests and fix failing tests in src/. Do not edit tests." ``` Do not combine a permission profile with `--sandbox`: OpenAI says the two systems "do not compose". **Cursor** Cursor documents run modes for the same decision; set them in the editor per your register row. For scripted runs, the `@cursor/sdk` options `local.autoReview` and `local.sandboxOptions` make the posture explicit in code. The full example is in [permissions and sandboxing](/en/workflows/harness/permissions-and-sandboxing/). ## How do you prove the human's decisions still hold? Nobody on a Level 4 team reads every diff, so the decisions above need evidence of their own. Run these four checks, and have the register owner sign off on them each quarter: | Check | What goes red | Where it runs | | --- | --- | --- | | Oracle protection | An agent-authored pull request changes files under `tests/`, `evals/` or the CI config without a code-owner approval | Branch protection with **Require review from Code Owners** | | Permission drift | A settings file, sandbox profile or allowlist changes without a matching register update | Code-owner review on `.claude/`, `.codex/` and `.cursor/` | | Unreviewed merges | The share of merged pull requests with no review rises in a risk class that requires review | A weekly query against your Git host, per risk class | | Expired decisions | A register row's review date has passed | A scheduled job that reads the register | Unreviewed merges are the leading signal. Faros AI's AI Engineering Report 2026 (April 2026, telemetry from 22,000 developers) found 31.3% more pull requests merging with no review at all. Nobody decided that; it happened one busy afternoon at a time. What you read instead of the diff is covered in [reading evidence instead of code](/en/ladder/evidence-not-diffs/). ## Copy-paste prompts for keeping the human's job **Copy-paste prompt: write the architecture brief the agents must obey** ```text Read this repository and draft the architecture brief my agents must not violate. Output four sections and nothing else. Invariants: the boundaries, layering rules and dependency directions this codebase actually enforces today. Cite a file for each one. Earned exceptions: the places where the code breaks its own rule on purpose, and the reason, from git history or comments. Off limits: the modules where a change requires a human decision, with globs. Open questions: the three architectural decisions that are currently implicit and would be expensive to get wrong, phrased as questions for me to answer. Do not propose refactors. Do not edit files. ``` **Copy-paste prompt: sort the backlog into loops that may run unattended** ```text Take the open issues in this backlog and sort them into three buckets using the autonomy ledger: runs unattended, human reads the plan, human reads the diff. For each issue, answer the four questions in one line each: what oracle decides done; can the agent edit that oracle; how long until a wrong answer surfaces; what the blast radius is. Then give the bucket and the single check that would move it one bucket safer. Flag separately any issue whose input includes text an outsider can write (issue bodies from external reporters, webhook payloads, scraped content). Output a table sorted by bucket. Do not start work on any of them. ``` **Copy-paste prompt: design the permission set an unattended run gets** ```text Design the permission policy for unattended runs in this repository, for each of the three tools I use (Claude Code, Codex, Cursor). For each tool state: the mode an unattended run must use; the commands allowed without asking; the commands that must always prompt or be blocked outright; the network domains reachable; the paths writable and the paths read-only; and the secrets that must not be present in the environment at all. Derive every entry from what the repository actually needs. Read the package scripts, CI config and any existing settings files first, and list, at the end, every permission you granted that you could not justify from a file. Do not edit any settings file; output the policy as a table. ``` ## Where does the human's job quietly disappear, and how do you get it back? **The reviewer becomes an approver.** Approval stops being a decision and becomes a click. To recover, write down which risk classes may merge on evidence alone, require review for the rest in branch protection, and track the unreviewed-merge share per class. **The oracle becomes the spec.** When tests are the only statement of what "correct" means, product decisions migrate into fixtures where nobody reviews them. To recover, keep acceptance criteria in the spec, map each one to the check that proves it, and review changes to that mapping like code; see [writing specs at Level 4](/en/ladder/level-4-write-specs/). **Accountability drifts to the tool.** "The agent did it" is not an answer in an incident review. To recover, add a named accountable person to every register row before the loop runs again, as Linear does with its human assignee. **Permissions drift through convenience.** Someone widens an allowlist to unblock one run, and every later run inherits it. To recover, put settings files under code-owner review and treat every change as a register update with a reason. **Understanding goes last.** A line Karpathy quotes approvingly in his published transcript is the warning: "You can outsource your thinking, but you can't outsource your understanding." A team that can no longer explain its own system has lost something no throughput number shows. To recover, have a human explain each architectural change in the pull request description, and rotate who does it; [craft and career](/en/ladder/craft-and-career/) covers which skills to keep in practice. **Level 5 gets mistaken for the destination.** Shapiro's dark factory, where "humans are neither needed nor welcome", describes "a handful of people" in teams of fewer than five. The six responsibilities above describe Level 4, which is where the published evidence sits. ## Where to go next with the human's job Before this page, read [one map of the ladder](/en/ladder/one-map/) for where each loop sits. Next, read [reading evidence instead of code](/en/ladder/evidence-not-diffs/) for what replaces the diff. - [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) — What a human reads when nobody reads every diff. - [Human judgment at the gates](/en/method/principles/human-in-the-loop/) — The four gates where a person decides, run on one task. - [Governance and autonomy](/en/org/governance-autonomy/) — Risk classes, guardrails and gates, and an autonomy policy template. - [Permissions and sandboxing](/en/workflows/harness/permissions-and-sandboxing/) — The per-tool configuration behind every permission decision. - [Level 5: You Run the Software Factory](/en/ladder/level-5-run-the-factory/) — Six stations, and which of them may run unattended. - [The State of Agentic Engineering](/en/ladder/state-of-agentic-engineering-2026/) — The sourced evidence behind the numbers on this page. --- # Vibe Coding vs Agentic Engineering: 21 Terms Compared URL: https://developertoolkit.ai/en/ladder/vibe-coding-vs-agentic-engineering/ Description: 21 AI-development terms set against vibe coding: who named each and when, who writes the code, who checks it, and how it differs. Every quote dated and linked. Vibe coding, introduced by Andrej Karpathy on 2 February 2025, means accepting AI-written code without reading it. Set against it are 20 other names for working with AI in software development: disciplines, methods, lifecycles, techniques and a tool category. Each is compared on who writes the code and what checks it. A vendor deck promises AI-driven development. The CTO's memo asks for agentic engineering. The policy bans vibe coding, and the engineer shipping the feature calls it AI-assisted coding. Four names are in play, and two of them say nothing about who checks the code. This page is for the tech lead or CTO who has to write one of these names into a policy, and for the developer who has to explain the difference. ## What you'll walk away with from these 21 definitions - One table that places 21 names by kind, origin and how each differs from vibe coding, each linked to its own definition page - The origin of each name with its date, and the "coined by" claims the dated sources do not support - A verbatim contrast with vibe coding for each name, or a label saying the contrast is our analysis - Our placement of vibe coding on the autonomy ladder, with the nearest primary evidence - Three copy-paste prompts that classify how your own repository's loops are verified ## How do the 21 names differ from vibe coding? Dates are the earliest dated use the research found, not proven firsts. Every quote below links to the page it comes from. Where a source offers no contrast with vibe coding, the contrast is labelled "our reading". | Name | Kind of thing | Named or popularised by | How it differs from vibe coding | | --- | --- | --- | --- | | [Vibe coding](/en/glossary/vibe-coding/) | Technique | Andrej Karpathy, 2 Feb 2025 | The anchor | | [Vibe engineering](/en/glossary/vibe-engineering/) | Discipline (a name that lost) | Simon Willison proposed it, 7 Oct 2025 | Same idea as agentic engineering, with "vibe" kept on purpose | | [Augmented coding](/en/glossary/augmented-coding/) | Personal practice | Kent Beck, 18 Apr 2025 | You care about the code, not only the behaviour | | [Agentic engineering](/en/glossary/agentic-engineering/) | Discipline | In use from 2025; endorsed by Karpathy, 4 Feb 2026 | Same agents, opposite stance on accountability | | [Agentic coding](/en/glossary/agentic-coding/) | Tool capability | No single coiner; earliest use found, Anthropic, 21 Jun 2024 | Names what the tool does, not how carefully you use it | | [AI-assisted coding](/en/glossary/ai-assisted-coding/) | Umbrella category | No coiner; DORA 2025 report title, 23 Sep 2025 | The superset: vibe coding is its "less responsible subset" (Willison) | | [AI pair programming](/en/glossary/ai-pair-programming/) | Interaction framing | GitHub Copilot, 29 Jun 2021 (earlier: Codota, 2017) | The human reads every suggestion; vibe coding reads none | | [Agentic software development](/en/glossary/agentic-software-development/) | Lifecycle | No coiner; Microsoft's "agentic DevOps", 19 May 2025 | Scope is the whole lifecycle, not one session | | [Agentic SDLC (AI SDLC)](/en/glossary/ai-sdlc/) | Lifecycle and operating model | No coiner; used by Gartner, Forrester and Anthropic in 2026 | Vibe coding has no stages and no gates | | [AI-DLC](/en/glossary/ai-dlc/) | Methodology (AWS) | AWS, Raja SP, 31 Jul 2025 | Built to stop the "accept all" habit (our reading) | | [AI-native development](/en/glossary/ai-native-development/) | Organisational paradigm | Tessl, Guy Podjarny, 6 Jul 2024 | A property of a team's process, not one person's review habit | | [AI-driven development](/en/glossary/ai-driven-development/) | Generic adjective | No owner; AWS's "AI-Driven Development Lifecycle", 31 Jul 2025 | "AI-driven" says who leads; vibe coding says whether anyone reads | | [Spec-driven development](/en/glossary/spec-driven-development/) | Method | No single AI-era coiner; Kiro, 14 Jul 2025; GitHub Spec Kit, 2 Sep 2025 | Vendors position it as the step after vibe coding | | [Compound engineering](/en/glossary/compound-engineering/) | Method and plugin | Kieran Klaassen, Every, 18 Aug 2025 | Every treats vibe coding as a fast path inside its system | | [Prompt engineering](/en/glossary/prompt-engineering/) | Technique | No coiner; in use around GPT-3 in 2020 | Different axis: the quality of the input, not review of the output | | [Context engineering](/en/glossary/context-engineering/) | Discipline | Popularised by Tobi Lütke, 19 Jun 2025, and Karpathy, 25 Jun 2025 | Vibe coding curates nothing; context engineering curates what the agent reads | | [Harness engineering](/en/glossary/harness-engineering/) | Discipline | Mitchell Hashimoto, 5 Feb 2026; an OpenAI post popularised it, 11 Feb 2026 | Replaces reading the output with automated checks | | [AI coding agents](/en/glossary/ai-coding-agents/) | Tool category | No coiner | The tool; vibe coding is one way to use it | | [Agentic workflows](/en/glossary/agentic-workflows/) | Design approach | Popularised by Andrew Ng, Mar 2024 | Different axis: checking inside the loop, not removed after it | | [Agent orchestration](/en/glossary/agent-orchestration/) | Technique and tool category | No coiner; Anthropic's orchestrator-worker pattern, 13 Jun 2025 | Not the opposite of vibe coding; verification is | | [AI engineering](/en/glossary/ai-engineering/) | Discipline and job role | Popularised by swyx, 30 Jun 2023 | Different axis: what is built, not how the code is written | The last column is our reading, except for AI-assisted coding, spec-driven development and compound engineering, where it follows a source's own contrast (quoted in each name's section below). ## What is vibe coding, and which names grew up around it? Three names come from the original post and the argument that followed it: the term itself, the proposed replacement, and Kent Beck's name that began as a synonym and became its contrast. ### What is vibe coding? Andrej Karpathy introduced the term in an [X post on 2 February 2025](https://x.com/karpathy/status/1886192184808149383): "There's a new kind of coding I call 'vibe coding', where you fully give in to the vibes, embrace exponentials, and forget that the code even exists." The same post sets the review rule: "I 'Accept All' always, I don't read the diffs anymore." Collins Dictionary named it its [Word of the Year for 2025](https://blog.collinsdictionary.com/language-lovers/collins-word-of-the-year-2025-ai-meets-authenticity-as-society-shifts/) on 6 November 2025. Merriam-Webster's [dictionary entry](https://web.archive.org/web/20260918165742/https://www.merriam-webster.com/dictionary/vibe%20coding) (Wayback capture of 18 September 2026) defines it broadly as "the act or practice of using an artificial intelligence system to generate computer code". Simon Willison keeps the narrow sense: ["building software with an LLM without reviewing the code it writes"](https://simonwillison.net/2025/Mar/19/vibe-coding/) (19 March 2025). This page uses the narrow sense. Willison's post gives the coinage date as 6 February, but the original post is timestamped 2 February 2025 (23:17 UTC), and Merriam-Webster's etymology note also says February 2, 2025. See the [vibe coding glossary page](/en/glossary/vibe-coding/). **Who writes the code, who checks it:** The model writes; nobody reads the diff, the human checks behaviour by eye. ### What is vibe engineering? Simon Willison [proposed the name on 7 October 2025](https://simonwillison.net/2025/Oct/7/vibe-engineering/) for "the other end of the spectrum, where seasoned professionals accelerate their work with LLMs while staying proudly and confidently accountable for the software they produce". He did not invent the words: Sergey Tselovalnikov titled a post ["There is no Vibe Engineering"](https://serce.me/posts/2025-31-03-there-is-no-vibe-engineering) on 31 March 2025, and Brendan O'Leary's talk "From Vibe Coding to Vibe Engineering" is [dated 29 April 2025](https://web.archive.org/web/20250513133609/https://boleary.dev/talks/vibe-engineering/). Willison's contrast: vibe coding is "the fast, loose and irresponsible way of building software with AI—entirely prompt-driven, and with no attention paid to how the code actually works." On 23 February 2026 he added to the same post: "It looks like the term 'Agentic Engineering' is coming out on top for this now." Addy Osmani gave the reason on 4 February 2026: ["the word 'vibe' carries too much baggage. It signals casualness."](https://addyosmani.com/blog/agentic-engineering/) Our reading: the name that lost the naming contest to agentic engineering, for the same idea. See the [vibe engineering glossary page](/en/glossary/vibe-engineering/). **Who writes the code, who checks it:** Agents write; the accountable professional checks. ### What is augmented coding (AI-augmented development)? Two names share a word and should not be merged. Kent Beck's earliest use of "augmented coding" found is from [18 April 2025](https://newsletter.kentbeck.com/p/augmented-coding-negative-feedback): "I have been doing a lot of vibe coding (which I call 'augmented coding' for reasons that will become clear)." By [25 June 2025](https://newsletter.kentbeck.com/p/augmented-coding-beyond-the-vibes) he drew the line: "In vibe coding you don't care about the code, just the behavior of the system. If there's an error, you feed it back into the genie in hopes of a good enough fix. In augmented coding you care about the code, its complexity, the tests, & their coverage." Gartner's separate category, AI-augmented development, is in its [press release of 16 October 2023](https://www.gartner.com/en/newsroom/press-releases/2023-10-16-gartner-identifies-the-top-10-strategic-technology-trends-for-2024): "the use of AI technologies, such as GenAI and machine learning, to aid software engineers in designing, coding and testing applications." See the [augmented coding glossary page](/en/glossary/augmented-coding/). **Who writes the code, who checks it:** The "genie" writes; the programmer checks through tests and design. ## Which names sit closest to agentic engineering? Four names sit closest to agentic engineering, at different scopes: a discipline, a tool capability, an umbrella and an interaction style. Only the discipline names a quality bar. ### What is agentic engineering? The phrase was in use before Andrej Karpathy endorsed it. Mario Zechner wrote of ["what people call 'agentic engineering'"](https://mariozechner.at/posts/2025-06-02-prompts-are-code/) on 2 June 2025, and Peter Steinberger titled a post ["Just Talk To It - the no-bs Way of Agentic Engineering"](https://steipete.me/posts/just-talk-to-it) on 14 October 2025. [Karpathy called it](https://x.com/karpathy/status/2019137879310836075) "personally my current favorite" on 4 February 2026 and stated the goal: "to claim the leverage from the use of agents but without any compromise on the quality of the software". Willison [defines it](https://simonwillison.net/2026/Feb/23/agentic-engineering-patterns/) as building software using coding agents, "where the defining feature is that they can both generate and execute code—allowing them to test that code and iterate on it independently of turn-by-turn guidance from their human supervisor" (23 February 2026). Osmani's contrast, [4 February 2026](https://addyosmani.com/blog/agentic-engineering/): "Vibe coding = YOLO. Agentic engineering = AI does the implementation, human owns the architecture, quality, and correctness." The shortest version is in Karpathy's published transcript of his [Sequoia Ascent talk](https://karpathy.bearblog.dev/sequoia-ascent-2026/) (30 April 2026; the page says an LLM produced the summary and cleaned up the transcript, and that Karpathy read them): "Vibe coding raises the floor. Agentic engineering is about extrapolating the ceiling." See [the state of agentic engineering](/en/ladder/state-of-agentic-engineering-2026/) and the [agentic engineering glossary page](/en/glossary/agentic-engineering/). **Who writes the code, who checks it:** Agents write; the human owns quality, tests and evals check. ### What is agentic coding? No single coiner is established. The earliest dated use the research found is Anthropic's [Claude 3.5 Sonnet announcement](https://www.anthropic.com/news/claude-3-5-sonnet) of 21 June 2024, which cites "an internal agentic coding evaluation". Anthropic made it a product category with Claude Code on [24 February 2025](https://www.anthropic.com/news/claude-3-7-sonnet): "a command line tool for agentic coding". An academic definition, from [Sapkota, Roumeliotis and Karkee](https://arxiv.org/abs/2505.19443) (26 May 2025): "agentic coding enables autonomous software development through goal-driven agents capable of planning, executing, testing, and iterating tasks with minimal human intervention." [Hassan et al.](https://arxiv.org/abs/2509.06216) draw the line to the discipline: "elevating the practice from agentic coding to true agentic software engineering" (7 September 2025). Our reading: agentic coding and vibe coding are not opposites. Karpathy's original vibe coding ran in an AI editor ("Cursor Composer w Sonnet"), and nothing stops anyone from accepting everything an agentic coding tool does. Agentic coding says what does the work; vibe coding says how much you look. See the [agentic coding glossary page](/en/glossary/agentic-coding/). **Who writes the code, who checks it:** An agent in a write-run-test loop; the term is silent on checking. ### What is AI-assisted coding (AI-assisted development)? It is an umbrella with no coiner. DORA's 2025 report is titled ["State of AI-Assisted Software Development"](https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report) (23 September 2025), and Wikipedia's article, in its revision of 5 October 2026, defines the field as ["the use of large language models (LLMs) and AI agents to assist software developers in software development"](https://en.wikipedia.org/wiki/AI-assisted_software_development). Willison's tag for it reads: ["Using AI tools such as Large Language Models to help write code. Vibe coding is the less responsible subset of this."](https://simonwillison.net/tags/ai-assisted-programming/) He explains why the name never worked as a brand: "I've tried in the past to get terms like AI-assisted programming to stick, with approximately zero success." And what falls outside vibe coding, [in his 2025 definition](https://simonwillison.net/2025/Mar/19/vibe-coding/): "If an LLM wrote the code for you, and you then reviewed it, tested it thoroughly and made sure you could explain how it works to someone else that's not vibe coding, it's software development." The autonomy ladder on this site uses "assisted" more narrowly, for Level 1. See the [AI-assisted coding glossary page](/en/glossary/ai-assisted-coding/). **Who writes the code, who checks it:** Human, AI or both; the term does not say who checks. ### What is AI pair programming? GitHub Copilot popularised the phrase in its [launch post of 29 June 2021](https://github.blog/news-insights/product-news/introducing-github-copilot-ai-pair-programmer/), "Introducing GitHub Copilot: your AI pair programmer". It did not invent it: Codota's homepage already read ["Your AI Pair Programmer"](https://web.archive.org/web/20170615194135/https://www.codota.com/) in a Wayback capture of 15 June 2017. The critique comes from inside the practice. Birgitta Böckeler wrote on martinfowler.com on [10 August 2023](https://martinfowler.com/articles/exploring-gen-ai/05-not-your-pair-programmer.html): "The framing of coding assistants as pair programmers is a disservice to the practice". Microsoft's agent announcement of [19 May 2025](https://azure.microsoft.com/en-us/blog/agentic-devops-evolving-software-development-with-github-copilot-and-microsoft-azure/) says the agent "levels up GitHub Copilot from a pair programmer to peer". On the ladder, pairing is Level 2: [Dan Shapiro](https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/) writes, "You are pairing with the AI like a colleague." Our reading: the human reads every suggestion as it appears, which is the opposite end of review from vibe coding. See the [AI pair programming glossary page](/en/glossary/ai-pair-programming/). **Who writes the code, who checks it:** The human drives and the AI suggests; the human reads every suggestion. ## Which names describe lifecycles and methods? Seven names describe how work moves from intent to release, from a vendor's lifecycle to a one-person loop. ### What is agentic software development? Forrester's Diego Lo Giudice, on [8 June 2026](https://www.forrester.com/blogs/agentic-software-development-takes-the-lead-from-code-assistants-to-orchestrated-sdlc-agents/): "The new norm is agentic software development, where autonomous agents collaborate across the entire software development lifecycle (SDLC), toward more end-to-end automation." No single coiner is established. Microsoft wrote of ["agentic DevOps"](https://azure.microsoft.com/en-us/blog/agentic-devops-evolving-software-development-with-github-copilot-and-microsoft-azure/) on 19 May 2025, and [Hassan et al.](https://arxiv.org/abs/2509.06216) of "Agentic Software Engineering (SE 3.0)" on 7 September 2025. Forrester puts vibe coding at the front of the lifecycle, not in place of it: "Product managers/owners vibe prototypes and features for the rest of the team to productize." Our reading: the same agents at the opposite scope, a single person's session that ends when "it mostly works" (Karpathy) against an organisation's lifecycle that ends at a release with gates. See the [agentic software development glossary page](/en/glossary/agentic-software-development/). **Who writes the code, who checks it:** Agents across the lifecycle; pipelines, other agents and humans at gates. ### What are the agentic SDLC and the AI SDLC? Anthropic's [AI-native SDLC playbook](https://claude.com/blog/the-ai-native-sdlc-playbook) (Louis Claxton, 21 August 2026) treats the labels as one thing: "You'll also hear this shift called the agentic SDLC, the AI SDLC, or simply agentic software development — the labels differ, but they describe the same thing." This page keeps agentic software development as its own entry because it has its own glossary page. The playbook's definition: "The AI-native SDLC is a reimagined process that combines the old control objectives with new enforcement. Instead of a linear flow, the process becomes a loop, and AI is embedded at each point." Nobody owns the label; Gartner's [abstract of 23 June 2026](https://www.gartner.com/en/documents/8039033) speaks of leaders who "transition to an agentic SDLC". GEICO's Jet Anderson, writing about vibe-coded applications in a [post updated on 2 June 2026](https://www.geico.com/techblog/the-agentic-sdlc/): "The failure is not that security tools did not catch the bugs. The failure is that nobody ran any security tools at all." Our reading: vibe coding has no stages and no gates, and the agentic SDLC is the gates. This site's [AI-native lifecycle guide](/en/method/lifecycle/) lays out six stages, each committing an artifact. See the [AI SDLC glossary page](/en/glossary/ai-sdlc/). **Who writes the code, who checks it:** Agents write the first pass at each stage; humans own the gates. ### What is AI-DLC? AI-DLC is a named methodology from AWS. Raja SP introduced it on [31 July 2025](https://aws.amazon.com/blogs/devops/ai-driven-development-life-cycle/): "This is why we're introducing the AI-Driven Development Lifecycle (AI-DLC), a new methodology designed to fully ingrain AI capabilities into the very fabric of software development." It replaces sprints with "bolts" measured in hours or days and epics with Units of Work. AWS's 2025 description has three phases (Inception, Construction, Operations); the open-source workflow in [`awslabs/aidlc-workflows`](https://github.com/awslabs/aidlc-workflows) has five phases and 33 stages (README of v2.11.0, read on 10 October 2026). AWS does not use the words "vibe coding" in any of its three AI-DLC posts (searched on 10 October 2026). Its nearest contrast is with "AI-autonomous development, where AI is expected to generate entire applications without human intervention", and its [open-source announcement](https://aws.amazon.com/blogs/devops/open-sourcing-adaptive-workflows-for-ai-driven-development-life-cycle-ai-dlc/) of 29 November 2025 says the mob rituals "ensure that AI's suggestions are not blindly accepted". Our reading: AI-DLC is built to stop the "accept all" habit that defines vibe coding. See the [AI-DLC glossary page](/en/glossary/ai-dlc/) and [this site's AI-DLC comparison](/en/ecosystem/frameworks/enterprise-frameworks/). **Who writes the code, who checks it:** AI writes after the plan is approved; the team validates in mob sessions. ### What is AI-native development (AI-native engineering)? Tessl's Guy Podjarny popularised it for software on [6 July 2024](https://tessl.io/blog/announcing-tessl-the-ai-native-development-startup/): "We're looking to define a new software development paradigm, AI Native Software Development, in which AI is built-in, not bolted on." Patrick Debois systematised it in ["The 4 patterns of AI Native Dev"](https://tessl.io/blog/the-4-patterns-of-ai-native-dev-overview/) (19 March 2025); he did not coin it. Gartner adopted "AI-native software engineering" in a [press release of 1 July 2025](https://www.gartner.com/en/newsroom/press-releases/2025-07-01-gartner-identifies-the-top-strategic-trends-in-software-engineering-for-2025-and-beyond). OpenAI's [AI-native team guide](https://cdn.openai.com/business-guides-and-resources/building-an-ai-native-engineering-team.pdf) (PDF last modified 20 November 2025) puts the roles this way: "The agent becomes the first-pass implementer; the engineer becomes the reviewer, editor, and source of direction." The label proves little by itself. Nearform's Cian Clarke, on [Tessl's blog](https://tessl.io/blog/from-vibe-coding-to-spec-driven-development/) (6 January 2026): "Many companies think enabling Copilot means they're doing AI-native engineering. It doesn't work that way." Shapiro puts ["90% of 'AI-native' developers"](https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/) at Level 2. Our reading: AI-native is a property of a team's process and vibe coding is one person's review habit, so an AI-native team can vibe-code a prototype and still ship through gates. See the [AI-native development glossary page](/en/glossary/ai-native-development/). **Who writes the code, who checks it:** Agents implement first; engineers own review and merge. ### What is AI-driven development? It is a generic adjective with no owner and no canonical definition, and Wikipedia has no article for it (checked via its API on 10 October 2026). The strongest anchor is AWS's methodology name, AI-DLC, from 31 July 2025. AWS's [post](https://aws.amazon.com/blogs/devops/ai-driven-development-life-cycle/) splits the field in three: "AI-assisted development, where AI enhances specific tasks", "AI-autonomous development", and its own third way, in which "AI systematically creates detailed work plans, actively seeks clarification and guidance, and defers critical decisions to humans." McKinsey also writes of "AI-driven software development" in a [3 November 2025 article](https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/unlocking-the-value-of-ai-in-software-development) that advises: "Avoid weak proxies like the percentage of code generated by AI, which offer little insight into real productivity." No source contrasts it with vibe coding. Our reading: "AI-driven" says who leads, and vibe coding says whether anyone reads. A team can be AI-driven and disciplined, or AI-driven and careless, and the word alone does not tell a CTO which. See the [AI-driven development glossary page](/en/glossary/ai-driven-development/). **Who writes the code, who checks it:** AI leads; the term names no checker. ### What is spec-driven development? No single coiner is established for the AI era. The phrase has academic prior art in a [2004 paper](https://doi.org/10.1007/978-3-540-24853-8_12) by Ostroff, Makalsky and Paige. In the AI era, Kiro used it on [14 July 2025](https://kiro.dev/blog/introducing-kiro/) and GitHub's Spec Kit followed on [2 September 2025](https://github.blog/ai-and-ml/generative-ai/spec-driven-development-with-ai-get-started-with-a-new-open-source-toolkit/). Birgitta Böckeler's [definition](https://martinfowler.com/articles/exploring-gen-ai/sdd-3-tools.html) (15 October 2025): "Spec-driven development means writing a 'spec' before writing code with AI ('documentation first'). The spec becomes the source of truth for the human and the AI." The vendors that launched the term set it against vibe coding. GitHub: "This 'vibe-coding' approach can be great for quick prototypes, but less reliable when building serious, mission-critical applications or working with existing codebases." Our reading: they position it as the step after vibe coding, not its enemy. This site's [spec-driven development guide](/en/workflows/plan/spec-driven-development/) is stricter than the industry definition: one authoritative `spec.md`, with a spec delta in every behaviour-changing pull request. See the [spec-driven development glossary page](/en/glossary/spec-driven-development/). **Who writes the code, who checks it:** Agents write; humans review the spec, plan and tasks; tests guard the result. ### What is compound engineering? Kieran Klaassen of Every named it on [18 August 2025](https://every.to/source-code/my-ai-had-already-fixed-the-code-before-i-saw-it), first as "compounding engineering": "building self-improving development systems where each iteration makes the next one faster, safer, and better." Dan Shipper co-wrote the [11 December 2025 article](https://every.to/chain-of-thought/compound-engineering-how-every-codes-with-agents), by which time it was called compound engineering; he did not coin the term. The core idea: "each unit of engineering work should make subsequent units easier—not harder." Every does not set it against vibe coding. Its [guide](https://every.to/guides/compound-engineering) (undated; the same section appears in the 2025 article) gives vibe coding a place inside the system: "Vibe code to discover what you want, then spec to build it properly." Our reading: vibe coding optimises the current change, and compound engineering optimises the next one by writing each lesson into files the next run reads. See [this site's compound engineering guide](/en/ecosystem/frameworks/compound-engineering/) and the [compound engineering glossary page](/en/glossary/compound-engineering/). **Who writes the code, who checks it:** Agents write; review agents check, then a human. ## How do prompt, context and harness engineering stack under the work? Alexey Grigorev's [July 2026 taxonomy](https://aishippingblog.com/p/ai-native-development-specifications) gives each layer its own question: "Prompt engineering - what we say when we interact with the agent", "Context engineering - what the agent knows before it starts". Its remaining layers are loop engineering and graph engineering. Our reading: harness engineering asks what constrains and checks the agent. ### What is prompt engineering? No single coiner is established. OpenAI's CLIP paper, submitted [26 February 2021](https://arxiv.org/abs/2103.00020), already refers to "the 'prompt engineering' discussion around GPT-3". Karpathy's [24 January 2023 post](https://x.com/karpathy/status/1617979122625712128) made the idea popular: "The hottest new programming language is English". OpenAI's [documentation](https://platform.openai.com/docs/guides/prompt-engineering) defines it as "the process of writing effective instructions for a model, such that it consistently generates content that meets your requirements." [Gartner's summary](https://www.gartner.com/en/documents/6781234) of 28 July 2025 framed it as displaced: "Context engineering is in, and prompt engineering is out." No source sets it against vibe coding. Our reading: they sit on different axes. Prompt engineering is about the quality of the input; vibe coding is the absence of review of the output. A prompt that names its own check stops being vibe coding. See the [prompt engineering glossary page](/en/glossary/prompt-engineering/). **Who writes the code, who checks it:** The model writes from the prompt; the human reads the output. ### What is context engineering? Tobi Lütke and Andrej Karpathy popularised it; neither coined it. Ankur Goyal [used it on 20 April 2025](https://x.com/ankrgyl/status/1913766591910842619), and Cognition's Walden Yan [on 12 June 2025](https://cognition.ai/blog/dont-build-multi-agents). Lütke, [19 June 2025](https://x.com/tobi/status/1935533422589399127): "I really like the term 'context engineering' over prompt engineering. It describes the core skill better: the art of providing all the context for the task to be plausibly solvable by the LLM." Karpathy, [25 June 2025](https://x.com/karpathy/status/1937902205765607626): "+1 for 'context engineering' over 'prompt engineering'." Anthropic's [definition](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) (29 September 2025): "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference". Sources disagree on the relationship to prompting. Anthropic calls context engineering "the natural progression of prompt engineering"; [LangChain](https://blog.langchain.com/the-rise-of-context-engineering/) argues "prompt engineering is a subset of context engineering". The nearest contrast with vibe coding is Thoughtworks' sponsored piece in [MIT Technology Review](https://www.technologyreview.com/2025/11/05/1127477/from-vibe-coding-to-context-engineering-2025-in-software-development/) (5 November 2025): "the transition from vibe coding to what's being termed context engineering shows that while the work of human developers is evolving, they nevertheless remain absolutely critical." Our reading: vibe coding curates nothing, while context engineering curates what the agent reads and says nothing about reviewing what it writes. See the [context engineering glossary page](/en/glossary/context-engineering/). **Who writes the code, who checks it:** The agent writes; the term is silent on checking. ### What is harness engineering? In the earliest use of the exact phrase the research found, Mitchell Hashimoto wrote on [5 February 2026](https://mitchellh.com/writing/my-ai-adoption-journey): "I've grown to calling this 'harness engineering.' It is the idea that anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again." He added: "I don't need to invent any new terms here; if another one exists, I'll jump on the bandwagon." Six days later an [OpenAI post](https://openai.com/index/harness-engineering/) by Ryan Lopopolo put the phrase in its title and the term spread; Wikipedia records the attribution as contested. LangChain's [definition](https://blog.langchain.com/the-anatomy-of-an-agent-harness/) (11 March 2026): "A harness is every piece of code, configuration, and execution logic that isn't the model itself." The OpenAI post describes the job: "a software engineering team's primary job is no longer to write code, but to design environments, specify intent, and build feedback loops that allow Codex agents to do reliable work." Our reading: harness engineering replaces reading the output with automated checks. Context engineering decides what the agent sees; harness engineering decides what checks and constrains it. Böckeler [nests harness inside context engineering](https://martinfowler.com/articles/harness-engineering.html) (2 April 2026), while Wikipedia's ["Agent harness"](https://en.wikipedia.org/wiki/Agent_harness) article puts context engineering inside the harness; neither nesting is settled. Dex Horthy [argues it is not enough](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/wsff.md) (22 July 2026): "no amount of harness engineering or loopsmaxxing can solve what is fundamentally a model-training issue." See [the harness guide](/en/workflows/harness/) and the [harness engineering glossary page](/en/glossary/harness-engineering/). **Who writes the code, who checks it:** The agent writes; mechanical checks and agent reviewers check. ## How do coding agents, agentic workflows and orchestration relate? Anthropic's [definitions of 19 December 2024](https://www.anthropic.com/engineering/building-effective-agents) give the topology: "Workflows are systems where LLMs and tools are orchestrated through predefined code paths." Agents "dynamically direct their own processes and tool usage". Our reading: orchestration is the third step, several agents coordinated by a lead agent or a script. ### What are AI coding agents? It is a tool category with no coiner. Willison's guide defines it plainly: ["What are coding agents? They're agents that can both write and execute code."](https://simonwillison.net/guides/agentic-engineering-patterns/what-is-agentic-engineering/) (chapter undated; the guide started on 23 February 2026). Vendors rename the category: GitHub's [changelog of 1 April 2026](https://github.blog/changelog/2026-04-01-research-plan-and-code-with-copilot-cloud-agent/) says "Copilot cloud agent (formerly known as Copilot coding agent)". On this site's [tool comparison](/en/comparison/), Cursor, Claude Code and Codex are coding agents that run the same loop: read the codebase, plan a multi-file change, edit files, run the project's checks and retry until they pass. The [feature matrix](/en/comparison/feature-matrix/) shows all three with subagents, worktrees, skills, hooks, MCP, a headless mode, an SDK and scheduled runs (Claude Code and Codex cells checked on 26 September 2026, Cursor cells on 28 August 2026). Willison's contrast, from the same guide: "Vibe coding is more useful in its original definition - we need a term to describe unreviewed, prototype-quality LLM-generated code that distinguishes it from code that the author has brought up to a production ready standard." Our reading: a coding agent is the tool, and vibe coding is one way of using it. See the [AI coding agents glossary page](/en/glossary/ai-coding-agents/). **Who writes the code, who checks it:** The agent writes; its own test runs check, then a human. ### What are agentic workflows? Andrew Ng popularised the phrase in March 2024; no source shows he coined it. His letter of [20 March 2024](https://www.deeplearning.ai/the-batch/how-agents-can-improve-llm-performance/) says "agent workflows". The [27 March 2024 letter](https://www.deeplearning.ai/the-batch/agentic-design-patterns-part-2-reflection/) is the first to use and define the phrase: "Instead of having an LLM generate its final output directly, an agentic workflow prompts the LLM multiple times, giving it opportunities to build step by step to higher-quality output." It is wider than software development, and coding agents are one application of it. Do not confuse it with GitHub's product, [GitHub Agentic Workflows](https://github.blog/changelog/2026-02-13-github-agentic-workflows-are-now-in-technical-preview/), in technical preview from 13 February 2026. No source contrasts it with vibe coding, and Ng's letters predate that term. Our reading: an agentic workflow moves the checking into the loop (reflection, tests), while vibe coding removes the human check after the loop. See the [agentic workflows glossary page](/en/glossary/agentic-workflows/). **Who writes the code, who checks it:** The model iterates; reflection, tests or a critic agent check. ### What is agent orchestration (multi-agent development)? "Orchestration" is an old systems word, and no single coiner is established for the agent sense. Ng's [17 April 2024 letter](https://www.deeplearning.ai/the-batch/agentic-design-patterns-part-5-multi-agent-collaboration/) lists multi-agent collaboration as a design pattern. Anthropic's [research-system post](https://www.anthropic.com/engineering/multi-agent-research-system) of 13 June 2025 describes "an orchestrator-worker pattern, where a lead agent coordinates the process while delegating to specialized subagents that operate in parallel", and cautions that "most coding tasks involve fewer truly parallelizable tasks than research". Per this site's feature matrix, Claude Code, Codex and Cursor all have subagents (checked on 26 September 2026 for Claude Code and Codex, 28 August 2026 for Cursor). One orchestrator's author links it to vibe coding directly. Steve Yegge on Gas Town, in [Welcome to Gas Town](https://steve-yegge.medium.com/welcome-to-gas-town-4f25ee16dd04) (1 January 2026): "It's also 100% vibe coded. I've never seen the code, and I never care to, which might give you pause." Our reading: orchestration multiplies whatever review discipline the human has. With a judge, tests and a merge gate it is a software factory; without them it is vibe coding at fleet scale. Orchestration is not the opposite of vibe coding, verification is. See [orchestration patterns](/en/workflows/automate/orchestration-patterns/) and the [agent orchestration glossary page](/en/glossary/agent-orchestration/). **Who writes the code, who checks it:** Worker agents write; a judge agent and deterministic checks, then the human at merge. ## Why is AI engineering a different job from all of these? The trap in this vocabulary is a name that sounds like the rest and means something else. ### What is AI engineering? swyx (Shawn Wang) popularised the role in ["The Rise of the AI Engineer"](https://www.latent.space/p/ai-engineer) on 30 June 2023: "The emerging (and least cringe) version of this role seems to be: AI Engineer." He did not start it: "I am calling attention to this trend rather than starting it." Chip Huyen's O'Reilly book, first released on 4 December 2024, defines the field as ["the process of building applications with readily available foundation models"](https://huyenchip.com/books/). So AI engineering is a job about what gets built: AI inside the product. The loose sense is the trap. Every's Klaassen [used "AI engineering"](https://every.to/source-code/my-ai-had-already-fixed-the-code-before-i-saw-it) on 18 August 2025 to mean AI-assisted coding: "Typical AI engineering is about short-term gains. You prompt, it codes, you ship." Our reading, since no source compares the two: AI engineering describes what is built, and agentic engineering describes how any software gets built with agents. They overlap only when an AI engineer uses coding agents, and a vibe coder may build an app with no AI inside it. See the [AI engineering glossary page](/en/glossary/ai-engineering/). **Who writes the code, who checks it:** Human engineers write; evaluation pipelines check the product. ## Where does vibe coding sit on the autonomy ladder? Dan Shapiro's ladder does not say. His [post of 23 January 2026](https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory/) never uses the word "vibe", Simon Willison's [write-up of 28 January](https://simonwillison.net/2026/Jan/28/the-five-levels/) does not either, and Karpathy's Sequoia Ascent page never mentions levels or the dark factory (full-text checks on 10 October 2026). The placement below is our analysis, built from the nearest primary sources. | Source | What it says | Date | | --- | --- | --- | | Karpathy, original post | "I 'Accept All' always, I don't read the diffs anymore." Its stop condition: "it mostly works" | 2 Feb 2025 | | Simon Willison, on the Level 5 team | A team "implementing what Dan Shapiro called the Dark Factory level of AI adoption, where no human even looks at the code the coding agents are producing" ([source](https://simonwillison.net/2026/Feb/7/software-factory/)) | 7 Feb 2026 | | StrongDM, the team Willison names | "Code must not be reviewed by humans"; validation uses scenarios "often stored outside the codebase (similar to a 'holdout' set in model training)", and "We use the term satisfaction to quantify this validation" ([source](https://factory.strongdm.ai/)) | 6 Feb 2026 | | Simon Willison | "as the coding agents get more reliable, I'm not reviewing every line of code that they write anymore, even for my production level stuff." ([source](https://simonwillison.net/2026/May/6/vibe-coding-and-agentic-engineering/)) | 6 May 2026 | Our reading: vibe coding and Level 5 share one trait, that no human reads the diff. They differ in what replaces the reader. In vibe coding nothing does; the human watches behaviour until "it mostly works". At Level 5 a machine-checkable verification layer does: held-out scenarios and a satisfaction score. Vibe coding is a Level 5 workflow with the oracle removed. Willison's 2026 remark that the two "have started to blur" is the same point from the other side: the boundary is not whether anyone reads, it is what replaced the reading. Two of the 21 names map onto the ladder directly. AI pair programming is the Level 2 rung, and agentic engineering is the discipline that earns each step above it (our reading). The ladder's destination, the software factory, is Level 5, covered in [Level 5: you run the software factory](/en/ladder/level-5-run-the-factory/) and on the [software factory glossary page](/en/glossary/software-factory/). ## Copy-paste prompts for placing your own work on this map Run these against a repository you ship from. Each works unchanged in Claude Code, Codex and Cursor, and each ends with a report you verify by opening three of the files or pull requests it cites. **Copy-paste prompt: classify how this repository's loops are verified** ```text Read this repository's CI configuration, agent instruction files (AGENTS.md, CLAUDE.md, Cursor rules), test and lint scripts, and the last 30 merged pull requests. List the recurring classes of change that agents make here (for example dependency bumps, test repair, feature work). For each class, answer four questions with file paths or pull request numbers as evidence: who writes the change, what automated check could fail if the change is wrong, who reads the diff, and what stops the run. Then place each class on this scale: L0 AI only autocompletes; L1 human writes and AI does discrete tasks; L2 human and AI pair and the human reads every line; L3 AI writes and a human reviews diffs; L4 AI writes, tests decide and the human writes specs; L5 nobody reads the code and a machine-checkable layer replaces the reader. Mark any class where nobody reads the diff and no check could fail as vibe coding. Do not edit any files. ``` **Copy-paste prompt: find the changes that merged on vibes** ```text From the last 30 merged pull requests in this repository, list every one that merged with no human review comment and with no test, type gate, lint rule or validator run that could have failed. For each, say what kind of change it was and which single check (a test, a type gate, a lint rule or a held-out scenario) would have failed had the change been wrong. Propose the smallest version of that check and where it would run: a hook, CI or a stop condition. Report only; do not change any file. ``` **Copy-paste prompt: audit a policy's vocabulary** ```text Read docs/ai-policy.md (or the policy file I name). Find every phrase that names a way of working with AI in software development, such as vibe coding, agentic engineering, AI-assisted coding, AI-driven development, AI-native, AI engineering or spec-driven development. For each phrase, quote the sentence, say what it states about who writes the code, what checks it and who reads the diff, and flag it when the sentence does not say. Propose replacement wording that names the check instead of the label. Do not edit the file. ``` ## What breaks when a team renames vibe coding? These are our analysis of failure patterns, not measured rates. Each has a recovery step. - **Renaming vibe coding as agentic engineering without adding checks.** The new name changes what the work is called, not what happens to the diff. The OpenSSF glossary (last modified 20 March 2026) draws the line on review: ["If there is human review of the AI-generated code, then it is not vibe coding."](https://glossary.openssf.org/vibe-coding/) A team that accepts every agent change unread is vibe coding whatever its policy says. Karpathy's published transcript is blunt about the standard: "You are not allowed to introduce vulnerabilities because of vibe coding." Recovery: write the check that would fail when the code is wrong before you rename the loop. - **Adopting a lifecycle's rituals without its verification.** AWS describes the failure its rituals exist to prevent: "Developers, excited by automation, often drift into passive execution—allowing AI to 'decide everything.'" Bolts and Units of Work are vocabulary; the Mob Elaboration and Mob Construction sessions are where suggestions are not blindly accepted. The same holds for an AI SDLC diagram: stages without gates are labels. Recovery: name the gate and the person who owns it for every stage. - **Treating the boundary as fixed.** On 6 May 2026 Willison wrote: "Weirdly though, those things have started to blur for me already, which is quite upsetting." and "There's an element of the normalization of deviance here". A team that reads fewer lines each month without adding checks drifts toward vibe coding while keeping its vocabulary. Recovery: record, for each loop, what replaced the reader: a test, a validator or a held-out scenario. - **Repeating a coinage claim the dates contradict.** Zechner and Steinberger used "agentic engineering" before Karpathy endorsed it. Willison proposed "vibe engineering" after earlier uses. Goyal used "context engineering" before Lütke and Karpathy. Hashimoto disclaims coining "harness engineering", and swyx says of AI engineering that he was "calling attention to this trend rather than starting it". Recovery: write "popularised", "endorsed" or "used", and link the dated source. - **Reading "AI engineering" as engineering with AI.** The role and the book mean building products on foundation models. A policy that says "AI engineering" must say which sense it means, because the two readers it addresses are different people. ## Where to go next with the vibe coding vocabulary - [The AI coding glossary](/en/glossary/) - [The state of agentic engineering](/en/ladder/state-of-agentic-engineering-2026/) - [The autonomy ladder](/en/ladder/) - [Level 5: you run the software factory](/en/ladder/level-5-run-the-factory/) - [Reading evidence instead of code](/en/ladder/evidence-not-diffs/) - [The human's job in agentic engineering](/en/ladder/the-humans-job/) - [The AI-native software development lifecycle](/en/method/lifecycle/) - [Prototype policy: graduate by risk and evidence](/en/cto-scorecard-guide/team-vibe-coding-policy/) - [A–Z reference: AI development terms](/en/reference/glossary/) ---