Level 5: You Run the Software Factory

Level 5 of the autonomy ladder is the software factory: a pipeline where agents pull work from a queue, build and verify it, and open pull requests that no human wrote. It has six stations (intent, harness, loop, graph, verification and release), and a loop may run unattended only when an oracle the agent cannot edit decides it is done.

This page is for the tech lead or platform engineer who already runs Level 4 loops and for the CTO deciding whether to fund a factory pilot. The typical situation: your /goal runs finish overnight, the review queue is now the slowest part of delivery, and someone asks whether the agents could merge dependency bumps on their own. This page answers which loops can, what each station needs first, and what Claude Code, Codex and Cursor ship for it.

How close is anyone to a dark factory today?

Dan Shapiro named this rung in The Five Levels: from Spicy Autocomplete to the Dark Factory (23 January 2026): “It’s a black box that turns specs into software”, where “humans are neither needed nor welcome”. He places only “a handful of people” there, in teams of “less than five people”.

The best-documented production case, Stripe’s Minions, keeps a human in the loop. Stripe reported that “over 1,300 Stripe pull requests” a week are “completely minion-produced, human-reviewed, but containing no human-written code” (Stripe, 19 February 2026). The agents run against “over three million” existing tests and get “at most two rounds of CI” (Stripe, 9 February 2026) before the branch goes back to a human. The proof layer predates the agents.

The cost of skipping that layer is also measured. Faros AI’s telemetry from 22,000 developers (AI Engineering Report 2026, April 2026) shows task throughput per developer up 33.7% and PR merge rate per developer up 16.2%, while incidents per pull request rose 242.7%, median time in review rose 441.5%, and 31.3% more pull requests merged with no review. So treat Level 5 as a property of individual loops, never of a whole company.

What are the six stations of a software factory?

Each station owns one decision. A station that emits no evidence is one you supervise by reading.

Station	Decision it owns	Evidence it must emit	Symptom when it is missing
Intent	What the agent knows before it starts	A spec with acceptance criteria and a named stop condition	Plausible code that solves the wrong problem
Harness	What the agent may run and reach	A permission and sandbox policy under version control	A run that works because it had more access than it should
Loop	When the agent stops	A machine-checkable stop condition and a retry cap	Runs that never finish or finish by declaring victory
Graph	Who does what when there are several agents	A written topology: planner, workers, judge, isolation	Agents overwriting each other’s files or context
Verification	What counts as proof	Deterministic checks the run cannot edit, plus review findings	Green checks on tests the agent rewrote
Release	How work leaves the building	An evidence bundle on the pull request and a named merger	A merge nobody can explain a week later

The stations map onto the lifecycle and the ladder on the one-map page. The deeper build guide, with the full autonomy ledger, is software factories.

Which loops may run unattended?

A level belongs to a loop, not to a team. Four questions decide whether a loop may run dark, and a loop must pass all four:

1. What oracle decides “done”? Name the specific check. “The test suite and the type checker pass” is an oracle. “It looks right” is not.
2. Can the agent fake it? If the agent can edit the oracle, the oracle is a suggestion. Tests inside the diff fail this question by construction.
3. How long until a wrong answer surfaces? Minutes is a pass. “A customer notices in three weeks” is a fail.
4. What is the blast radius if it is wrong? Reversible and contained is a pass. A schema migration or a payment path is a fail.

Adopt this table as the rule your team applies before any loop leaves supervision:

Loop class	Oracle	Agent can edit it?	Time to surface	Blast radius	Verdict
Patch and minor dependency updates	Full CI suite	No, if CI config and tests are protected	Minutes	One revertable commit	Runs unattended, human merges
Lint and codemod migrations	Linter plus type checker	No, if rule config is protected	Minutes	Mechanical, revertable	Runs unattended, human merges
Flaky-test quarantine	Re-run statistics from CI history	Yes, unless quarantine lives outside the repo	Days	Hides real failures	Supervised
Feature work from a spec	Acceptance tests written with the spec	Yes, if they live in the same diff	Days to weeks	Product behaviour	Supervised, evidence review
Auth, money, schema, migrations	Partial at best	Varies	Weeks	Irreversible	Stays with a human reading code

The last row is the short list of change classes where someone still reads the code; reading evidence instead of code explains why.

Copy-paste prompt: find the first loop that can run dark

Read the last 60 merged pull requests in this repository (use `gh pr list
--state merged --limit 60 --json number,title,files,labels`) and group them
into loop classes: repeated kinds of change such as dependency updates, lint
fixes, flaky tests, copy changes or feature work.
For each class, answer the autonomy ledger in a markdown table:
1. the oracle that decides "done", as a command that exits 0;
2. whether an agent working on that class could edit that oracle, citing
   the file paths it would have to touch;
3. how long a wrong change of that class took to surface, using reverts and
   follow-up fixes in the git log as evidence;
4. the blast radius if it is wrong.
End with the single class that passes all four, or say that none does and
name the one change that would make the best candidate pass.
Do not modify any files.

How do you stand up the first factory loop?

Build one loop at a time, harness first. The steps use dependency updates, the table’s first row, because its oracle already exists.

1. Write the intent as a file in the repository, for example .factory/dep-bump.md, so every run gets the same instructions and the prompt goes through code review like any other change. Use the prompt below as its content.

2. Put the oracle out of the agent’s reach. Add a CODEOWNERS entry for tests/, .github/ and the lint and type configs, and turn on “Require review from Code Owners” in the branch ruleset; a CODEOWNERS file on its own only requests reviewers. Make the CI checks required, and add the CI step from the failure modes below that fails any factory branch touching protected paths. The run can still change code, but it cannot merge a weaker test. Protecting the oracle covers hooks and holdout tests.

3. Fix the harness. Run the agent in a disposable CI job or cloud environment with an explicit allowlist, a spending cap and no secrets except the model credential. Check out with persist-credentials: false so no write token sits in .git/config while the agent runs. Opening the pull request is a separate job that runs no agent: the agent job uploads its changes as a patch artifact, and the second job, with its own narrowly scoped token (contents: write, pull-requests: write), applies the patch, pushes and opens the pull request.

4. Run the loop headless with the tool’s non-interactive mode (commands below). The loop’s stop condition is in the prompt: at most two fix attempts, which mirrors Stripe’s two CI rounds.

5. Release with evidence, not with a diff. The pull request carries the run summary, the exit codes of the three checks and the dependency’s changelog link. A human merges from that evidence bundle and reads code only when the evidence is incomplete.

6. Measure before you widen. Track revert rate, incidents per pull request and review wait for the loop for four weeks. Add a second loop class only when those three are flat or better.

Copy-paste prompt: the build station for dependency updates

You are the build station of a dependency-update loop. Scope: one outdated
direct dependency per run, patch or minor versions only. If the only update
available is a major version, stop and report it.
1. Run `npm outdated --json`, pick the patch or minor update that has been
   available longest, and update only that package and the lockfile.
2. Run `npm test`, `npm run lint` and `npm run typecheck`. If any fails, fix
   the application code, never the tests, the lint config or the type config.
   Make at most two fix attempts, then stop.
3. Do not edit anything under tests/ or .github/, or any lint or tsconfig file.
4. Finish with a JSON summary: package, from, to, each of the three commands
   with its exit code, the files you changed, and "status": "ready" or
   "needs-human" with a one-sentence reason.

Each headless command runs from the repository root in the CI job or disposable runner from step 3.

Claude Code

dontAsk mode auto-denies every tool call that would otherwise prompt, so the run never waits for input: only file reads, read-only commands and the tools on the allowlist run.

claude -p "$(cat .factory/dep-bump.md)" \
  --permission-mode dontAsk \
  --allowedTools "Read" "Edit" "Bash(npm outdated *)" "Bash(npm install *)" \
    "Bash(npm test)" "Bash(npm run lint)" "Bash(npm run typecheck)" \
  --max-budget-usd 5 \
  --output-format json > .factory/last-run.json

For scheduled runs without your own CI, a routine runs the same prompt as a cloud session on a schedule, from an API call or from a GitHub event. Routines have no permission-mode picker and no approval prompts during a run, so the allowlist lives in the environment, not in the session.

Codex

codex exec needs no approval flag: it has no -a of its own and already runs with approval never, so a command outside the sandbox fails instead of waiting for input (checked on CLI 0.157.1). The workspace-write sandbox restricts network access unless you enable it, and this loop needs the registry.

codex exec \
  --sandbox workspace-write \
  -c sandbox_workspace_write.network_access=true \
  -o .factory/last-run.md \
  "$(cat .factory/dep-bump.md)"

Legacy --sandbox still works in 0.157.1; OpenAI now prefers permission profiles (beta, -c default_permissions=":workspace"). Do not combine a profile with --sandbox until re-verified, because OpenAI says the two systems do not compose.

For recurring runs, Codex automations schedule the same task in the background. In CI, openai/codex-action@v1 wraps the same non-interactive run.

Cursor

Cursor’s CLI has a print mode (-p, --print) and runs in GitHub Actions, but for a factory loop the more direct route is an Automation with a Scheduled or Source control trigger that starts a Cloud Agent with .factory/dep-bump.md as its task. Cloud Agents run in isolated VMs, and the Cloud Agents API sends a statusChange webhook when a run finishes, so your systems learn the result without polling (Cursor documentation, checked 2026-08-28).

Caution

Never add --dangerously-skip-permissions, bypassPermissions or Codex’s --dangerously-bypass-approvals-and-sandbox to a factory loop outside a disposable, network-restricted sandbox. A loop that needs full access to pass has told you its harness is wrong.

How do you know factory output is right without reading every diff?

• Deterministic gates decide, models advise. Types, tests, lint and fail-closed migration checks decide whether a change may merge. Review agents such as Claude Code’s /code-review, Codex’s /review and Cursor’s Bugbot add findings, but none of them is deterministic, so none may be the only gate.
• The oracle lives outside the diff. Required checks come from a workflow the factory branch cannot change, either a ruleset “Require workflows to pass” pinned to the default branch or a pull_request_target job (a pull_request_target job must not check out or execute the branch’s code with secrets in scope; run the branch’s tests in a separate unprivileged job), and a CI step fails any factory branch that touches .github/. A pull_request workflow runs from the branch’s own merge ref, so a branch that edits it changes its own check. CODEOWNERS with required code-owner review covers test and config paths, and a holdout suite the agent never sees catches overfitting to visible tests.
• Every pull request carries an evidence bundle: the intent file, the commands and exit codes, the review findings and the run log. The agent PR review page covers reading one.
• A sampling audit replaces line-by-line review. Each week, a named owner reads a small random sample of merged factory pull requests end to end. If the sample finds a defect the gates missed, that loop goes back to supervised until a new check covers the gap.
• Four numbers per loop: revert rate, incidents per pull request, review wait and share merged with no human approval.

The loop’s owner signs off on the loop, and the merger signs off on each pull request from its evidence. Nobody signs off on “the factory” as a whole; that is the human’s job of reading the seams.

Copy-paste prompt: write the verification contract for one station

Station: verification. Loop: patch and minor dependency updates.
Write the station's contract as a markdown table with one row each for: the
inputs it accepts; the deterministic validators that must pass before
hand-off, each as a command that exits 0; whether the agent can edit any of
those validators, and if it can, a replacement that lives outside the diff;
the failure route, naming which station receives a FAIL and what it gets
back; the maximum retries before the run stops; and the human escalation
trigger. Then answer the autonomy ledger for this loop: what oracle decides
done, can the agent fake it, how long until a wrong answer surfaces, and
what the blast radius is. End with a one-line verdict: runs unattended, or
stays supervised and why.

What does each tool ship for each station?

Claude Code rows, including their “Not documented” cells, were re-checked on 2026-09-26 against code.claude.com and the installed CLI (2.1.283). Codex rows were checked against CLI 0.157.1 on 2026-09-26 where the CLI shows them, and against OpenAI’s documentation on 2026-08-28 otherwise. Cursor rows were last verified on 2026-08-28, because cursor.com was unreachable on 2026-09-26. “Not documented” is a gap in the documentation on that date, not proof the tool cannot do it. Re-check the vendor page on the day you build against it.

Intent: context, procedure and plan

Capability	Claude Code	Codex	Cursor
Persistent project instructions	CLAUDE.md and auto memory	AGENTS.md	Rules
Reusable procedure	Skills	Skills (/skills)	Agent Skills
Distributable bundle	Plugins	Plugins (codex plugin)	Plugins
Plan before implementing	Plan mode (/plan)	Plan mode (/plan)	Plan Mode
Guidance on writing a verifiable objective	Write an effective /goal condition: end state, stated check, constraints	Long-running work: outcome, constraints, verification	Not documented (2026-08-28)

Both vendors describe the same three parts: the outcome, the check that proves it and what must not change. Copy them into every intent file, whichever tool runs it.

Harness: permissions, sandbox and compute

Capability	Claude Code	Codex	Cursor
Permission policy	Permission modes: Manual, acceptEdits, plan, auto, dontAsk, bypassPermissions	Approval policies untrusted, on-request (default), granular, never; permission profiles (beta)	Run modes
Approvals decided by a model	Auto mode classifier	Auto-review (--approve-for-me)	Auto-review for local tool calls
Operating-system sandbox	Sandboxed Bash (macOS, Linux, WSL2)	--sandbox modes and codex sandbox	/sandbox in the CLI
Lifecycle hooks	Hooks	Hooks (need persisted trust)	Hooks
External tools	MCP	MCP (codex mcp)	MCP
Self-hosted compute	Self-hosted environments (Team and Enterprise beta)	Not documented (2026-08-28)	Private workers in the Cloud Agents API
Push external events into a live session	Channels (research preview)	Not documented (2026-08-28)	Not documented (2026-08-28)

Claude Code’s classifier blocks actions that escalate beyond the request, target unrecognized infrastructure or appear driven by hostile content: a judgement made by a model. For an unattended loop, prefer an explicit allowlist and treat any classifier as a second line.

Loop: stop conditions, schedules and headless runs

Capability	Claude Code	Codex	Cursor
Goal or stop condition	/goal	/goal (stable, on by default)	/goal (rolling out, 2026-08-19)
Interval loop	/loop	No /loop (checked 2026-09-26, CLI 0.157.1)	/loop bundled skill
Scheduled unattended runs	Routines (research preview, one-hour minimum)	Automations	Automations
Headless invocation	claude -p	codex exec	Print mode (-p)
Programmatic SDK	Claude Agent SDK	Codex SDK	@cursor/sdk, cursor-sdk
Isolated cloud run	Claude Code on the web	Codex cloud	Cloud Agents

The triggers differ more than the runners: routines start on a schedule, an API call or a GitHub event; Codex’s automations named Gmail, Slack and GitHub pull request activity (2026-08-28); Cursor’s automations add Webhook, Linear, Sentry and PagerDuty. If your factory pulls from a queue, that trigger list is the queue.

Graph: many agents without collisions

Capability	Claude Code	Codex	Cursor
Delegated subagents	Subagents, 20 running per session by default	Subagents (/subagents)	Subagents, on their own VMs since 2026-08-19
Peer agents that message each other	Agent teams (experimental, off by default)	Not documented (2026-08-28)	Not documented (2026-08-28)
Script-driven orchestration	Dynamic workflows	Not documented (2026-08-28)	Not documented (2026-08-28)
Filesystem isolation	Worktrees (--worktree)	Worktrees (--worktree, /worktree)	Worktrees
One screen for many agents	Agent view (claude agents, research preview)	codex agents command center	Agents window

Dynamic workflows “orchestrate many subagents from a script Claude writes and you can rerun”: a factory floor you can version. Orchestration patterns covers the planner, workers and judge topology in all three tools. The general name for coordinating several agents like this is agent orchestration.

Verification: review and rollback

Capability	Claude Code	Codex	Cursor
Review the local diff	/code-review	/review, codex review	Not documented (2026-08-28)
Deep multi-agent review	Ultrareview: /code-review ultra or claude ultrareview	Not documented (2026-08-28)	Not documented (2026-08-28)
Managed pull request review	Code Review (research preview, Team and Enterprise)	@codex review on GitHub and GitLab	Bugbot
Security review	/security-review, Claude Security plugin	Codex Security, @codex security review	Security Agents
Undo or rewind	Checkpoints (/rewind)	Not documented (2026-08-28)	/rewind in the CLI

Ultrareview goes furthest: “every reported finding is independently reproduced and verified”. A review typically takes 5 to 10 minutes and, after the free runs, costs $5 to $25 in usage credits, so it belongs on high-risk changes, not on every factory pull request.

Release: CI, triggers and the round trip

Capability	Claude Code	Codex	Cursor
Run in GitHub CI	anthropics/claude-code-action@v1	openai/codex-action@v1	Cursor CLI in GitHub Actions
Run in GitLab	GitLab CI/CD	Merge request review (@codex review)	GitLab integration
Trigger from your own systems	Routines API trigger	Not documented (2026-08-28)	Webhook triggers
Outbound notification of run status	Not documented (2026-09-26)	Not documented (2026-08-28)	Webhooks (statusChange)
Route and approve pull requests	Not documented (2026-09-26)	Not documented (2026-08-28)	PR Routing & Approval (vendor docs)
Steer a run from a phone	Remote Control	Codex Remote (codex remote-control, experimental)	Cloud Agents on mobile

The gap that matters is the round trip. A routine’s API trigger “returns a session URL”, but Claude Code documents no outbound webhook, so your systems poll. Cursor documents both directions.

Where a software factory rots first

The queue outruns review capacity. The symptom is review wait climbing week on week while throughput looks healthy, which is the Faros pattern. To recover, cap the number of open factory pull requests per loop, pause intake when the cap is reached, and move reviewers to the evidence bundle instead of the diff.

The oracle drifts into the agent’s reach. The symptom is a green run whose diff touches a test, a snapshot or a lint rule. To recover, revert the change, add the path to CODEOWNERS, and add a CI step that fails when a factory branch modifies protected paths.

Unattended runs inherit yesterday’s permissions. Routines and automations run without approval prompts by design, so whatever the harness allowed last month is what runs at 3 a.m. To recover, keep the allowlist in version control and review it like code. Set the permission mode in user or managed settings, because Claude Code ignores auto and bypassPermissions in project settings, and pin Codex’s limits in requirements.toml.

Untrusted text reaches an agent with tools. Issue bodies, pull request comments, webhook payloads and scraped pages are written by outsiders. To recover, give loops that read them read-only tools, run their untrusted-input steps with --bare --setting-sources "" --strict-mcp-config so no project hooks or MCP servers load (--bare authenticates only with ANTHROPIC_API_KEY or an apiKeyHelper, not an OAuth token), and hand the write step to a separate job. Claude Code already labels routine fire text as untrusted data.

Nobody owns the whole line. Six stations with six owners and no one reading the seams is how a green pipeline ships a wrong product. To recover, name one owner per loop who runs the weekly sampling audit and can switch the loop back to supervised without a meeting.

Where to go next at Level 5

Level 4: You Write the Specs The prerequisite: specs, stop conditions and hooks for one long run.

One map Ladder, lifecycle and factory stations in one grid.

Reading evidence instead of code What a human reads when the factory writes the diff.

Software factories The full build guide and autonomy ledger.

From issue to pull request A worked end-to-end factory loop in all three tools.

Shaping a backlog for agents Slice work into loops an agent can finish and prove.

Unattended agent runs The policy for bounded recurring work.

The human's job What no station can decide for you.