Software Factories: How to Build One and Keep It From Rotting

A software factory is a pipeline in which coding agents pull work from a queue, build and test it, and merge it with little or no human reading the diff. A factory holds up when each loop’s autonomy is capped by an oracle the team trusts more than the model, and rots when autonomy is declared factory-wide.

You run six agents. By Thursday there are forty pull requests, two people who can review them properly, and a queue that is now the team’s critical path. Faros AI’s AI Engineering Report 2026 (April 2026, two years of telemetry from 22,000 developers and more than 4,000 teams) measured the same pattern: median time to first review up 156.6%, and pull requests merged with no review at all up 31.3%.

This page is for the tech lead and developers who build that pipeline, and the CTO who decides how far it may run unattended.

What you’ll walk away with from designing a software factory

• Your team’s place on the factory spectrum, in autonomy ladder levels.
• The autonomy ledger: four questions that decide whether a loop may merge without a human reading the diff.
• A build order for the factory floor, including the context feed most pipelines leave out, with the Claude Code, Codex and Cursor pieces.
• The WIP rule and the metrics that detect rot before incidents do.
• Three copy-paste prompts: a readiness audit, a plan that carries production context, and an adversarial gate reviewer.

What is a software factory?

The loop has five steps:

1. Humans decide what to build and put it in a queue: issues, tickets, or a label.
2. Agents build and test a change that satisfies the requirement, then open it for CI and review.
3. Approved changes deploy to production through CD.
4. Rejected changes go back to an agent, which iterates until the gate passes or a human takes over.
5. Users engage with what shipped and emit errors, usage data and complaints that inform the next item in the queue.

The one map names the six stations, and Level 5: run the software factory lists what each tool ships for each. This page settles what those do not: how much autonomy each loop has earned, and how to keep the codebase healthy once it runs.

Where does your team sit on the factory spectrum?

Use two axes: how much of the code agents write, and what a human still reads before it merges. The published reference points:

Where	Agent-written share	What a human reads	Ladder level of the loop
Most teams	Assisted, human-authored pull requests	Every line	L1–L2
Stripe’s Minions	“Over 1,300 Stripe pull requests … merged each week are completely minion-produced, human-reviewed, but containing no human-written code” (Alistair Gray, Stripe’s engineering blog, 2026-02-19)	The diff, after at most two CI rounds against more than three million tests	L3
Dan Shapiro’s “dark software factory”	All of it	Nothing: “humans are neither needed nor welcome” (Shapiro, 2026-01-23). Simon Willison identifies the team as StrongDM’s AI division (2026-01-28)	L5

A loop that merges with nobody reading needs three things at once, and teams usually have one: real adoption (not just access), validation to the point of mergeability (independent tests, browser checks, a second reviewer, never an agent grading tests it wrote itself), and guardrails that assume failure (sandboxes, scoped credentials, deterministic checks).

Why do software factories erode, and what are the two remedies?

Two diagnoses shaped the 2026 discussion: one about the codebase, one about the pipeline. You need both remedies: feed the agent real evidence, and meter autonomy by what you can verify. The unit of decision is the loop, not the factory, because a factory is dozens of loops with different verification economics.

The erosion case: harness engineering is not enough

At AI Engineer World’s Fair in July 2026, HumanLayer’s Dex Horthy gave a talk titled “Harness Engineering is not Enough: Why Software Factories Fail”. According to the talk’s title and abstract (a secondary source; the talk has not been transcribed), its argument is that a better pipeline, better prompts and better tooling cannot on their own produce a codebase that stays maintainable at scale.

Our reading of why: an agent loop rewards passing the checks it can run, and nothing scores maintainability, because a design’s quality depends on requests nobody has made yet. Every change can be green while the codebase gets worse.

SlopCodeBench measured that directly (Orlanski et al., arXiv 2603.24755, v1 2026-03-25, v2 2026-05-07). Instead of single-shot tasks, it makes coding agents repeatedly extend their own earlier solutions, checkpoint after checkpoint, as the specification evolves. It tracks two signals over the whole trajectory: verbosity (redundant or duplicated code) and structural erosion (complexity concentrating in a few functions). The headline result: no agent solves any problem end to end, and the best agent passes 14.8% of checkpoints.

The slope has to be caught outside the agent: a human reading plans, or a scheduled measurement of the codebase.

The context case: the pipeline starves the agent

This is our view, drawn from how published factories are wired rather than from a named study: the pipeline, not the model, is what starves the agent.

Most published factories separate deciding what to build from building it: a human plans, compresses the result into a ticket, and hands it to an agent that has never seen the problem. A sound design needs the constraints and the evidence: how the feature is used, which customers hit the bug, what the stack trace says. An agent handed a ticket always works from the summary, and tends to write new code where a better-informed engineer would change existing code.

How do you decide which loops may run unattended?

Grant a loop as much autonomy as you can cheaply and reliably verify, and no more.

The autonomy ledger

Before a loop merges without a human reading the diff, answer four questions about it in writing:

1. What oracle decides “done”? Name the command. “Tests pass” is an oracle. “It looks right” is not.
2. Can the agent fake it? If the agent can edit the oracle, the oracle is a suggestion. Tests the agent writes in the same change fail this question by construction.
3. How long until a wrong answer surfaces? Seconds is a green light. “A human notices in three weeks” is a red one, and it is the answer for every architectural decision.
4. What is the blast radius if it is wrong? Reversible and contained, or a schema migration.

A loop that answers cleanly on all four can merge by rule. A loop that fails question 2 or 3 keeps a human in it, however good your harness is. The ledger decides a loop’s level; the change’s risk class still caps what any single change may do, so a Level 5 loop escalates an auth change exactly as a Level 2 team would (see the risk-class table on the one map).

Loop	Oracle	Fakeable?	Feedback latency	Verdict
Dependency bumps, CVE patches	Build plus the full test suite across consumers	No, the suite predates the change	Minutes	L5: merges by rule
Flaky-test stabilization	Run the test 1,000 times and measure the failure rate	Yes, the fix edits the test itself	Minutes	L4: a human reads the diff of the test file. L5 only if a diff guard proves the assertion lines are unchanged
Mechanical codemods, lint and type fixes	Compiler and linter	Only if suppressions and lint or tsconfig changes are blocked by a CI check that fails on any new eslint-disable, @ts-ignore or # noqa in the diff	Seconds	L5 with that check in place; L4 without it
Feature work in a well-covered module	Existing tests plus new ones the agent wrote	Yes, the agent owns both sides	Minutes	L4: a human reads the plan and the evidence
Schema, auth, pricing, migrations	Production behavior	No, but slow	Days to weeks	L3: a human reads the diff
Architectural refactors	None that runs in CI	Not applicable	Weeks to months	L3, with the planning front-loaded

Copy-paste prompt: factory-readiness audit

Audit this repository for agentic factory readiness and produce an autonomy
ledger. Do not propose a pipeline yet, and do not change any files.

Step 1. Inventory the classes of change this repo actually receives. Run
`git log --since="6 months ago" --name-only --format="%h %s"` and group
commits by what they touched (dependency manifests, migrations, tests,
feature code by module), not by commit-message prefix. Report the share of
changed lines that landed in files that already existed versus new files.

Step 2. For each class of change, fill in a table with these columns:
  - oracle: the exact command that decides "done" (read CI config to find it)
  - fakeable: could an agent editing this repo weaken or delete that oracle
    in the same change? Answer yes if the tests live in paths the agent edits.
  - latency: how long after a wrong change merges before anything surfaces it
  - blast radius: what breaks, and whether it is reversible without a deploy

Step 3. Assign each class a verdict: merges by rule, a human reads the plan
and evidence, or a human reads the diff. Justify every "merges by rule" with
the answers from step 2, and flag any class whose only oracle is a test file
the agent would also be editing.

Step 4. For the highest-volume class that still needs a human, list the
missing infrastructure that would let it move one step toward merging by
rule, cheapest first. Name files and commands that do not exist yet; do not
recommend generic best practices.

How do you build the factory floor?

Build in this order. Steps 1 and 2 are the ones teams skip, and skipping them produces a fast pipeline that ships erosion.

1. Make the repository legible to an agent. An AGENTS.md or CLAUDE.md with the architecture and non-obvious constraints, typed module boundaries, and tests that pin behavior, which limit the damage a wrong agent decision can do. See making a codebase agent-ready.

2. Wire the context feed. Before an agent plans, it reads what an engineer would open: error traces, logs, usage data and the customer’s own words. Connect the error-tracking and analytics MCP servers you already pay for, and make reading them a required planning step. In Claude Code:

   # Terminal: remote servers with OAuth; the first tool call asks you to log in
   claude mcp add --transport http sentry https://mcp.sentry.dev/mcp
   claude mcp add --transport http posthog https://mcp.posthog.com/mcp

   Codex takes the same server with codex mcp add sentry --url https://mcp.sentry.dev/mcp followed by codex mcp login sentry, because the remote server authenticates with OAuth (checked with codex 0.157.1 codex mcp --help). In Cursor, add the same URL under mcpServers in .cursor/mcp.json ({"sentry": {"url": "https://mcp.sentry.dev/mcp"}}). Debugging production from the editor has the per-tool setup.

3. Build the queue. A label on an issue is enough. The queue makes the work list explicit and rate-limited, so what enters is a decision, not an accident. See from issue to pull request.

4. Build the gate in order of cost. Deterministic checks first: compiler, linter, type checker, test suite, contract tests. Then an agent reviewer prompted adversarially. Then a human, on the loops your ledger says need one. Running the expensive reviewer before the free checks burns tokens on diffs a type error would have rejected.

5. Sandbox, scope and meter. Each agent gets an isolated environment, credentials scoped to its loop, and a spend cap. See permissions, sandboxes and approval modes.

6. Instrument the factory itself. It is a production system whose output is code: track queue depth, review latency and rework rate per loop with agent observability.

Caution

Tests inside the repository are not a gate against the agent that can edit them. Keep the behavioral scenarios that decide “done” out of the agent’s reach, like a validation split in training: in a path the agent cannot write, a protected branch, or a separate repository. Treat any change that touches tests or CI configuration as high risk class, whatever the loop’s level. Protecting the oracle walks through the five layers of protection.

Which tool runs which station?

The three tools cover different surfaces, and most factories mix them. The split matches the one in continuous delivery with AI.

Claude Code

Claude Code is the scriptable station. claude -p runs headless in a CI job, which is how you wire a gate reviewer or a scheduled sweep. The flags carry the guardrails: --allowedTools limits what the agent may touch, --output-format json makes the result parseable, and --max-budget-usd caps the spend of one run (all checked against claude --help, v2.1.283).

# CI, on a pull request. The prompt file comes from the default branch,
# never from the contributor's checkout.
# trusted/ is a checkout of the default branch, not the contributor's.
git diff origin/main...HEAD > trusted/pr.diff
claude -p "$(cat trusted/.github/prompts/gate-review.md) Review trusted/pr.diff." \
  --bare --setting-sources "" --strict-mcp-config \
  --allowedTools "Read,Grep,Glob" \
  --output-format json \
  --max-budget-usd 2 > verdict.json
jq -r '.result' verdict.json | tail -n 1 | grep -q '^PASS' || exit 1

--bare --setting-sources "" --strict-mcp-config stops the job from loading hooks, settings or MCP servers from the pull request itself, which would otherwise run with the job’s secrets. --bare reads only ANTHROPIC_API_KEY (or an apiKeyHelper) for auth, never OAuth, so set that secret in the job (checked with v2.1.283). Headless agents in CI covers the full workflow file.

Pick Claude Code when the factory lives in .github/workflows.

Codex

Codex is the multi-surface station. codex exec runs a non-interactive worker that exits, codex review --base main reviews a branch from the shell, and on GitHub @codex review reviews a pull request. The Slack and Linear integrations let work enter the factory from where the team already talks (verified 2026-08-28).

# CI gate. The job needs one secret, OPENAI_API_KEY.
# trusted/ is a checkout of the default branch, not the contributor's.
printenv OPENAI_API_KEY | codex login --with-api-key
git diff origin/main...HEAD > trusted/pr.diff
codex exec -C trusted --sandbox read-only \
  -o verdict.txt \
  "$(cat trusted/.github/prompts/gate-review.md) Review pr.diff."
tail -n 1 verdict.txt | grep -q '^PASS' || exit 1

-C trusted makes the trusted checkout the agent’s working root, so the contributor’s AGENTS.md and .codex/ configuration are never loaded; only the diff crosses over, as a file. --with-api-key reads the key from stdin (checked with codex exec --help and codex login --help, 0.157.1). Headless agents in CI shows the full workflow.

Pick Codex when work enters from chat or a tracker.

Cursor

Cursor’s strength is the review station and event-driven builds. Bugbot reviews pull requests for bugs, security issues and code-quality problems. Cloud Agents run in isolated VMs with full development environments, and Automations start them on a schedule or on events from GitHub, GitLab, Slack, Linear, Sentry, PagerDuty or a webhook. A Sentry-triggered automation is the context feed wired in directly. PR Routing & Approval decides who must approve what (verified 2026-08-28). See Bugbot for setup.

Pick Cursor when production events should start agents without a script.

Copy-paste prompt: a plan that carries production context

Write the implementation plan for the issue below. Do not write code yet, and
do not trust the issue description as a complete account of the problem.

First, gather evidence with the connected tools:
- The related error-tracking issue: full stack trace, frequency, first seen,
  affected releases, and how many users hit it.
- Usage data for the surface involved: how often the path runs, the drop-off,
  and whether the behavior changed after a recent release.
- Every existing file that already implements this behavior. Report what
  share of the work extends existing code versus adds new files.
- Prior attempts in git history (`git log -S` on the key identifiers), and
  what happened to them.

Then write the plan. Open it with a section titled "What the evidence says
the problem is" that confirms or contradicts the issue description. If they
contradict, stop and say so instead of planning around it.

The plan must name the files it will change, the existing abstraction it will
extend rather than duplicate, and the specific command that will prove it
worked. If no such check exists, say so instead of inventing a test that only
checks the code you are about to write.

Issue: #1234

Replace #1234 with the issue number; the agent reads the issue through your tracker’s MCP server or gh issue view.

How do you run a factory without accumulating hidden debt?

Once the floor runs, three dials decide whether it stays healthy: how much work enters, what you measure, and what it costs.

Set the WIP limit by review capacity, not token budget

Adding agents widens the funnel’s mouth; with the gate unchanged, only the queue grows, and queues in front of gates turn into merges without gates. The Faros 2026 report shows it: median time in review (distinct from time to first review) up 441.5%, and pull request size up 51%.

Cap concurrent agents at the number of changes your gate can clear per day. Raise the cap only by making the gate faster or cheaper: moving a loop up the ledger, adding a deterministic check, or shrinking pull requests. Team parallelism covers calibrating the ceiling per developer.

Which metrics catch factory rot early?

Incident counts lag by months. Track these instead, split by agent-authored and human-authored changes. Faros’s data comes from its own customers’ telemetry, not a random sample, so read its column as a direction, not a benchmark.

Metric	What it tells you	Faros, AI Engineering Report 2026
Code churn (code deleted soon after it was written)	Throughput that is not real	Up 861%
Incidents per pull request	Whether the gate works	Up 242.7%
Bugs per developer	Defects escaping the gate	Up 54%
Pull requests merged without review	Whether the gate is skipped	Up 31.3%
Median time to first review	Where the queue forms	Up 156.6%
Duplication and complexity concentration	SlopCodeBench’s two erosion signals, measured on your repo	Not in the report; measure it weekly with the codebase health loop

Quote the same report in both directions. It also found epics completed per developer up 66.2%, task throughput up 33.7% and pull request merge rate up 16.2%, with deployments per week down 11.7%. The output is real, and so is the bill.

Meter cost per loop, not per seat

Autonomous loops spend without a human approving each run. DoltHub’s Tim Sehn put an orchestrated multi-agent session at about $100 in Claude tokens per hour, “about 10X the cost of a normal Claude Code session per unit time” (2026-01-15), and a week at $3,000 (2026-03-24): one user, one setup, but the order of magnitude. Cap each run (--max-budget-usd in claude -p) and review spend per merged change per loop. Cost governance covers the mechanics.

Fix the platform before you scale the factory

DORA’s 2025 report (2025-09-23) found AI adoption positively related to delivery throughput and negatively related to delivery stability. Its central theme: “AI doesn’t fix a team; it amplifies what’s already there.” The same report found “a direct correlation between a high quality internal platform and an organization’s ability to unlock the value of AI.”

What breaks in a software factory, and how do you recover?

Nobody can explain the code any more. The gap shows only when a bug needs someone to reason about the system. To recover, quarantine the area, write down the behavior you need from it as executable acceptance criteria, and regenerate from that spec rather than debugging code nobody owns. To prevent it, keep reading plans on loops where you have stopped reading diffs.

The agent edited the test instead of the code. The suite is green, the bug is in production, and the diff touched the assertion. That is ledger question 2 unanswered. Move the oracle out of reach, and treat any green run where a test file changed as an unreviewed change until a human signs it.

Erosion that no check catches. Every gate passes and the codebase still gets worse, because duplication and complexity concentration are invisible to compilers and test suites. Schedule the codebase health loop, and when its trend climbs for several weeks, move the affected loop one level down until a refactor lands.

Review theater. Approvals arrive seconds after the pull request opens, on diffs too large to have been read. A skimmed 800-line diff is worse than an unread one, because it leaves a record saying somebody looked. Review a random sample properly, cap pull request size in CI, and move the human’s reading to the evidence bundle; see reviewing agent pull requests.

Queue stalls. An agent picks up work and it sits. Alert on time in a state, and route anything past its threshold to a human instead of another retry.

Blast radius through paths you did not mediate. Your LLM gateway (the proxy in front of model calls) sees only the model calls routed through it, not direct API calls, shell commands or browser sessions. List the unmediated paths, because an autonomous loop with broad credentials is a different posture from an engineer with the same ones. See MCP security.

Copy-paste prompt: adversarial gate reviewer

Review this diff as the last automated gate before a human sees it. Assume
the author was an agent optimizing to make checks pass, and that it had
write access to the tests.

Report only findings you can tie to a specific failure. For each, state the
input or state that triggers it and what goes wrong. Skip style, naming and
anything a linter already covers.

Check, in this order:
1. Did this change weaken its own oracle? Flag any modified, deleted, skipped
   or loosened assertion, snapshot update, or CI config change, and say
   whether the production code justifies it.
2. Does this duplicate logic that already exists in the repo? Search before
   concluding it does not, and name the function it should have called.
3. What happens on the unhappy path: empty input, concurrent callers, partial
   failure, and a downstream timeout?
4. What is the blast radius if this is wrong in production, and can it be
   reverted without a deploy?
5. What did the author not touch that this change implies should also change?

End with one line: BLOCK or PASS, and if BLOCK, the single finding that
justifies it. Do not pad the list to look thorough.

Save this prompt as the gate-review.md file the tool tabs above read; the last line of each job fails the build unless the verdict is PASS. To check the reviewer itself, feed it a known-bad diff each month (one that loosens an assertion) and confirm it blocks.

Where to go next with software factories

Evidence instead of diffs The prerequisite: what a human reads when they stop reading every line

Level 5: run the software factory Station by station, what Claude Code, Codex and Cursor ship for each one

The one map The ladder, the lifecycle and the stations in one grid, with the risk-class table and a loop register

Keeping an agent-written codebase healthy The weekly erosion measurement and the thresholds that lower a loop's autonomy

Protecting the oracle Keep the tests that decide done out of the agent's reach

tmux for agent fleets The layer below the factory: seeing which agent is blocked and passing work between them