# AI Developer Toolkit — Strategy & business > The "Strategy & business" section of the AI Developer Toolkit documentation (English). Full site index: https://developertoolkit.ai/llms.txt Generated: 2026-11-02T15:41:09.219Z Source: https://developertoolkit.ai Total documents: 11 --- # Strategy: AI-native engineering for business leaders URL: https://developertoolkit.ai/en/strategy/ Description: The business decisions behind agentic engineering: economics, business case, org design, roadmap and board reporting, with the evidence standard behind them. The Strategy and business section covers the five decisions leadership owns when coding agents write most of the code: what an accepted change costs, whether to fund it at scale, how teams change shape, how fast to move, and what the board sees. Every number carries a publisher and a date. Your CTO wants budget for agents, or you are the CTO asking for it. A vendor promises a multiple, a board member forwards a study where developers got slower, and finance asks about headcount. This section gives you the unit, templates and gates to decide without reading code or taking a number on trust. ## Who this section is for, and where each reader starts | You are | Start with | Then | | --- | --- | --- | | CEO, CFO or COO | [The executive track](/en/start/executive/): a ten-minute brief and the five questions for your CTO | [Economics](/en/strategy/economics/), then [the business case](/en/strategy/business-case/) | | CTO or VP Engineering | [The CTO track](/en/start/cto/), which pairs these pages with the operating model in [Engineering organization](/en/org/) | [The transformation roadmap](/en/strategy/transformation-roadmap/), then [board reporting](/en/strategy/board-reporting/) | | Owner of a software house or agency | [AI-native delivery for software houses and agencies](/en/strategy/services-firms/) | [Economics](/en/strategy/economics/) for your own margin | Never measured where you stand? Take the free [C-Level Scorecard](/en/cto-scorecard/) first; it places you on the [autonomy ladder](/en/ladder/one-map/) these pages use. ## Which five decisions does this section settle? Each decision has one page and one artifact to adopt. | Decision | The question | What you adopt | Page | | --- | --- | --- | --- | | Economics | What does an accepted change cost us? | The cost-per-accepted-change definition and a spreadsheet template | [Economics](/en/strategy/economics/) | | Business case | Is it worth funding at scale? | A decision memo with four options, stop gates and a risk register | [Business case](/en/strategy/business-case/) | | Org design | How do teams, roles and headcount change? | Team sizing by review capacity and a headcount plan without multipliers | [Org design](/en/strategy/org-design/) | | Pace | How fast do we move, and what stops us? | A 12-month roadmap with stop/go gates per delivery loop | [Transformation roadmap](/en/strategy/transformation-roadmap/) | | Board reporting | What does the board see each quarter? | A one-page report with a five-row risk register | [Board reporting](/en/strategy/board-reporting/) | Supporting pages: [leading people through the change](/en/strategy/leading-the-change/), [product management when build time collapses](/en/strategy/product-management/), [software built outside engineering](/en/strategy/builders-beyond-engineering/), and [success stories and case studies](/en/strategy/success-stories/). The decisions are tool-agnostic; the economics and board-reporting pages show where Claude Code, Codex and Cursor keep cost data. Plan prices: [the pricing analysis](/en/comparison/pricing-analysis/); model prices: [the models hub](/en/ecosystem/models/). ## What evidence standard do these pages follow? The third-party evidence points both ways. DORA's 2025 report (Google Cloud, 23 September 2025) found "a positive relationship between AI adoption on both software delivery throughput and product performance" and "a negative relationship with software delivery stability". METR's randomized trial of 16 experienced open-source developers on early-2025 tools (10 July 2025) found they took 19% longer with AI, yet afterwards believed it had sped them up by 20%. Its follow-up (METR, 24 February 2026) favours AI on point estimates, but the intervals cross zero and METR calls the data an unreliable signal. The details are in [the state of agentic engineering](/en/ladder/state-of-agentic-engineering-2026/). So every page here follows the rule below; adopt it for your own decks. **Adopt as-is: evidence rule for AI engineering decisions** ```text EVIDENCE RULE FOR ANY AI ENGINEERING PROPOSAL OR REPORT 1. Every number names its publisher, its date and what was measured. 2. No productivity multiplier without a named third-party source and its population. 3. Vendor-internal figures are labelled "vendor-internal" and never used as a forecast. 4. Output (changes merged) is never shown without stability (change failure rate, incidents per change) and review load (hours in review). 5. Savings are counted in cost per accepted change: merged, not reverted within 30 days. Hours "saved" in a survey are not savings. 6. Every expected gain has a stop gate: the number that pauses the program, and the person who makes that call. ``` ## How leadership knows it works without reading the code No executive reviews a diff, and at scale neither can a CTO. Proof comes from machine-run checks and auditable numbers: - **The unit counts only changes that stayed merged.** Reverted work adds cost but leaves the denominator, so rework shows in the number. - **Autonomy is earned per loop, not granted by memo.** A team climbs only when its tests, checks and [evidence attached to each change](/en/ladder/evidence-not-diffs/) meet the gate in [autonomy governance](/en/org/governance-autonomy/). - **Sign-off is split.** The CTO owns metric definitions and finance signs the cost line. Every increase in agent autonomy is signed by the approver in [the roadmap's gate table](/en/strategy/transformation-roadmap/). The board sees the result on one page each quarter. | Autonomy step | Who signs | | --- | --- | | Up to L3 | Platform lead or loop owner | | L3 to L4 | CTO or a delegate, with the service owner | | L4 to L5 | CTO, with security sign-off | ## What goes wrong when leaders set AI engineering strategy? - **A seat rollout is reported as a transformation.** Seats measure spend. Report cost per accepted change and each critical loop's level instead. - **A multiplier reaches the board deck.** One question about its source loses the room. Use the break-even figure from [the business case](/en/strategy/business-case/). - **Headcount is cut before review capacity is known.** Review becomes the constraint and incidents rise. Size teams by review capacity from your own pilot, as [org design](/en/strategy/org-design/) shows. - **The roadmap follows the calendar, not the gates.** A quarter passes, so autonomy goes up. Tie each step to a stop/go gate and roll a loop back one rung when its stop signal fires ([the transformation roadmap](/en/strategy/transformation-roadmap/)). ## Where to go next with strategy - [The economics of agent-built software](/en/strategy/economics/) — Start here: the unit every page in this section uses - [The state of agentic engineering](/en/ladder/state-of-agentic-engineering-2026/) — The dated third-party evidence, both ways - [Designing a pilot that proves something](/en/org/pilot-design/) — The baseline and decision rule a business case needs - [Engineering organization](/en/org/) — The operating model, governance and security the CTO builds --- # Reporting AI engineering to the board URL: https://developertoolkit.ai/en/strategy/board-reporting/ Description: A quarterly one-page board report on AI engineering: autonomy level per critical loop, cost per accepted change, incident trend and a five-row risk register. A board report on AI engineering is a one-page quarterly update with four blocks: the autonomy level each critical delivery loop runs at, the cost per accepted change, the quality and incident trend, and a five-row risk register. It never uses lines of code, seat counts or pull-request volume as proof of value. The board approved the AI tooling budget a year ago. Now the audit committee chair asks two questions: what did the company get for it, and what could go wrong? You have a vendor dashboard with a rising pull-request count, a "62% of code is AI-written" slide from an engineering all-hands, and a case study from someone else's company. None of those survive the second follow-up question, and the chair knows it. This page is for the CTO who writes that report and the executive who presents or reads it. It is the last step of the executive and CTO reading tracks: it turns the [economics](/en/strategy/economics/), the [business case](/en/strategy/business-case/) and the [transformation roadmap](/en/strategy/transformation-roadmap/) into something the board can steer by. ## What you get from this board reporting kit - A one-page quarterly template you can paste into your board pack today. - Metric definitions for every number on the page, with the formula, the data source and the gaming risk. - Where each number comes from in Claude Code, Codex and Cursor, and what to do when a tool cannot supply it. - A risk register with five standing rows, each with an owner, a leading indicator and dated evidence. - A "what not to claim" table, with the published evidence behind each line. - A quarterly cycle with named sign-offs, two copy-paste prompts that draft and attack the pack, and the verification that runs before anything reaches the board. ## What goes on the one-page board report? Copy this into your board pack as-is. Every blank is a number you already have or a gap you declare as "not measured". A declared gap is acceptable in the first two quarters; an invented number never is. ```markdown # AI engineering: quarterly board update — Q_ YYYY Prepared by: CTO · Cost block co-signed by: CFO · Risk rows signed by: CISO, General Counsel Definitions version: v_ (link to the metric definitions; unchanged since last quarter: yes/no) ## 1. Headline (three sentences, no adjectives) - What changed: ___ - What it cost per accepted change, against last quarter: ___ - The one risk the board should watch: ___ ## 2. Autonomy level per critical loop | Loop (owner) | Level last Q | Level now | Target next Q | Gate that must hold to move up | Evidence link | |-------------------------|--------------|-----------|---------------|---------------------------------------|---------------| | Dependency updates (__) | L_ | L_ | L_ | e.g. full test suite + canary green | ___ | | Bug fixes, service A (__)| L_ | L_ | L_ | ___ | ___ | | Feature work, product B (__)| L_ | L_ | L_ | ___ | ___ | ## 3. Economics | Measure | Last Q | This Q | 4-quarter trend | Note | |-----------------------------------------|--------|--------|-----------------|------| | Total AI engineering spend vs budget | | | | | | Accepted changes (merged, not reverted) | | | | | | Cost per accepted change | | | | | | Share of spend that is review and rework| | | | | ## 4. Quality and incidents (split: agent-authored / human-authored) | Measure | Agent | Human | Trend | |-------------------------------------------|-------|-------|-------| | Change failure rate | | | | | Incidents per 100 accepted changes | | | | | Median time in review | | | | | Share of merges with no human or agent review | | | | | Median change size (lines) | | | | ## 5. Risk register (five standing rows) | Risk | Rating (L×I) | Trend | Leading indicator | Control in place | Owner | Next action | |-----------------|--------------|-------|-------------------|------------------|-------|-------------| | Security | | | | | CISO | | | IP and licensing| | | | | GC | | | Vendor | | | | | CTO | | | Skills pipeline | | | | | CTO/CHRO | | | Regulation | | | | | GC | | ## 6. Decisions requested 1. ___ (budget, a loop moving up a level, a policy approval) ## 7. What this report does not claim No productivity multiplier, no headcount saving, no "% of code written by AI" as value. ``` ## Which numbers go on the page, and how are they defined? Define every metric once, version the definitions, and change them only between quarters with a note in the header. A metric whose definition moves while the trend is being read is the fastest way to lose a board's trust. The canonical definitions of the AI metrics live on the [metrics frameworks page](/en/org/metrics-frameworks/); the table below is the board-level subset. | Metric | Definition | Source | Gaming risk and the counter-metric | | --- | --- | --- | --- | | **Level per critical loop** | The rung of the [autonomy ladder](/en/ladder/) (L0 to L5) at which a recurring unit of work runs, set by the first ladder question the loop fails | Loop owner's assessment plus the gate evidence | Inflating the level: require a named gate and a link to its last green run for every level above L2 | | **Accepted change** | A merged change that met its acceptance criteria and was not reverted within 30 days | Version control plus the revert log | Splitting work into tiny changes: report median change size beside it | | **Cost per accepted change** | (licences + metered usage + CI + review hours at loaded cost + rework + incident cost) ÷ accepted changes | Invoices, CI billing, time data; method on the [economics page](/en/strategy/economics/) | Leaving out review and rework: finance co-signs the inputs | | **Change failure rate** | Share of deployments that cause a failure needing remediation, split by provenance | Deploy and incident systems plus a provenance label | Relabelling incidents: the incident owner, not the author, sets the label | | **Incidents per 100 accepted changes** | Production incidents traced to a change, normalised by accepted changes | Incident postmortems | Under-reporting: count every Sev-1 and Sev-2 with a linked change | | **Median time in review** | Median time from review request to approval | Code host | Rubber-stamp approvals: report the share of merges with no review beside it | Two definitions do most of the work. "Accepted" makes the unit of value something the business recognises, and "split by provenance" is what lets the board see whether agent-authored work fails more or less often than human work. Provenance needs a label on every change; the [PR labelling guide](/en/cto-scorecard-guide/ai-pr-labeling/) sets one up that does not depend on any single vendor. If you also track human time, pair it with a guardrail. Anthropic's Claude Code team measures human time as human approvals per pull request (median and 95th percentile), human comments, and how long a pull request waits on a person. It treats revert and incident rates as the guardrail: "if they rise, we haven't saved human time, we've only moved it" (Anthropic engineering blog, 15 October 2026; a self-report about its own repositories). On this page's definitions, read median time in review beside the share of merges with no review (no human and no automated review), incidents per 100 accepted changes and the revert log behind "accepted change", and report a fall in human time only while those hold steady. ### How do you choose the critical loops? Pick three to six loops, not the whole engineering organisation. A loop qualifies when it recurs often enough to trend, when a failure in it reaches customers or regulators, and when an owner can name its gate. Typical first choices: dependency updates, bug fixes in a revenue-bearing service, feature work in the main product, and infrastructure changes. Report a level per loop, never one level for the company. The ladder itself says a level belongs to a loop: a dependency-update loop with a deterministic check can run at L4 while feature work beside it sits at L2, and both answers are correct at once. A single company-wide level hides exactly the difference the board needs to see. ## Where do the numbers come from in Claude Code, Codex and Cursor? The spend and adoption inputs differ by tool; the quality inputs do not. Change failure rate, incidents and review time come from your code host and incident system whatever the agent, so build those queries once. For deeper telemetry setup, see [agent telemetry](/en/ecosystem/observability/agent-telemetry/). **Claude Code** - **Analytics dashboard** (Team and Enterprise plans): active users, sessions and, with the GitHub app installed, pull requests with Claude Code and the share of merged pull requests with Claude Code. Merged pull requests with attributed lines carry the GitHub label `claude-code-assisted`, so this search on your code host gives a sample without API access: `is:pr is:merged label:claude-code-assisted`. - **Caveat for the board:** contribution metrics are a public beta and are unavailable under Zero Data Retention (Anthropic analytics docs, checked 2026-09-26). If you run ZDR, your own provenance label is the only source. - **OpenTelemetry export** for spend: set `CLAUDE_CODE_ENABLE_TELEMETRY=1`, `OTEL_METRICS_EXPORTER=otlp` and `OTEL_EXPORTER_OTLP_ENDPOINT` pointing at your collector (without an endpoint the exporter targets localhost and the board pack gets nothing; [agent telemetry](/en/ecosystem/observability/agent-telemetry/) has the full variable set); the `claude_code.cost.usage` metric feeds the economics block. - **Sanity check for the spend line:** Anthropic's cost documentation gives "around $13 per developer per active day and $150-250 per developer per month" for enterprise deployments (checked 2026-09-26). A figure far outside that range is worth a question before it reaches the board. **Codex** - **OpenTelemetry export:** an `[otel]` table in `~/.codex/config.toml` exports logs, traces and metrics to your collector (read from the Codex source for CLI 0.157.1, 2026-09-26). Send it to the same collector as your other agents so the economics block has one source. - **Local cost reports across agents:** `npx ccusage@latest codex monthly` reads local Codex logs and prints cost per month (ccusage 20.0.24, 2026-09-26). It covers one machine, so use it to check a vendor invoice, not to replace it. - **Enterprise analytics:** OpenAI describes a Codex analytics dashboard and API for ChatGPT Enterprise workspaces; its pages could not be read from the writing environment on 2026-09-26, so confirm with your admin which fields your workspace exposes before you put them in the template. **Cursor** - **Team analytics:** Cursor documents a team analytics dashboard and admin APIs for its team plans. cursor.com could not be reached from the writing environment on 2026-09-26, so confirm with your Cursor admin which fields your plan exposes before promising them to the board. - **Provenance:** do not depend on a per-commit AI attribution feature until you have confirmed your plan includes it. Label changes in your own pipeline instead; the label survives a tool switch. - **Spend:** reconcile the Cursor invoice with the accepted-change count from your code host; the formula is the same as for the other tools. With more than one agent, the rule holds for each: vendor dashboards describe adoption, your own pipeline describes outcomes, and the board page uses vendor data only for the spend line. ## How do you build the risk register? Keep five standing rows every quarter, even when a row is quiet. A board learns to trust a register that shows green rows turning amber; it stops trusting one where risks appear only after they have happened. | Risk | What the board needs to know | Leading indicator | Dated evidence you can cite | Canonical page | | --- | --- | --- | --- | --- | | **Security** | The agent and its toolchain are attack surface: prompt injection through issues and pull requests, stolen credentials, poisoned packages | Agent credentials without an expiry; workflows that let untrusted issue text reach an agent with write access | Cline's advisory of 2026-02-17 records an unauthorised npm publish of `cline@2.3.0`; the malicious `nx` releases (advisory 2025-08-27) collected credentials and posted them to GitHub | [Agent threat model](/en/org/agent-threat-model/) | | **IP and licensing** | Who owns agent-written code, licence contamination, and what vendor indemnities cover | Share of repositories with licence scanning in CI; client contracts silent on agent use | Checked with counsel each quarter, not from a statistic | [Legal and IP](/en/org/legal-and-ip/) | | **Vendor** | Models, defaults and plans change monthly, and a change can move quality or cost without any decision on your side | Loops whose evals have not been rerun since the last default-model change | Codex's bundled default became GPT-6 Astra on 2026-09-04 and GPT-6.1 Sol on 2026-09-29; Claude Code's default became Claude Opus 5.5 from v2.1.280 (the `latest` channel) on 2026-09-22; GPT-5.4 was retired in Codex on 2026-08-31 with automatic migration. Versions and prices live on the [models hub](/en/ecosystem/models/) | [Avoiding lock-in](/en/org/lock-in-and-portability/) | | **Skills pipeline** | Oversight needs the skills that heavy delegation erodes; junior engineers learn less by writing code | Juniors per senior reviewer; share of juniors' merged work they could explain in review | Anthropic's internal study (2025-12-02, a survey of 132 engineers and researchers): staff worry about "skills atrophying as [they] delegate more", and one respondent notes that "more junior people don't come to me with questions as often" | [Junior developers](/en/teams/junior-developers/) | | **Regulation** | Which duties reach a company that uses coding agents, and when shipping AI features changes that | AI-literacy training coverage; AI features shipped to EU users | After the Digital Omnibus (Regulation (EU) 2026/1744), EU AI Act Art. 50 transparency duties apply from 2 August 2026 and Annex III high-risk obligations are deferred to 2 December 2027 (secondary: Gibson Dunn and Usercentrics summaries; re-verify on EUR-Lex before the board meeting) | [EU AI Act](/en/org/eu-ai-act/) | Rate each row on likelihood and impact, show the trend arrow, and name the one action due before the next meeting. The vendor row is the one executives underestimate most: in September 2026 both Claude Code and Codex changed their default model within three weeks of each other, and a loop whose quality depends on the model can shift without anyone deciding to shift it. That is why the vendor row's indicator is "evals not rerun since the last default change". ### When does a risk skip the quarterly cycle? Agree escalation triggers with the chair in advance, so nobody has to judge in the moment whether something is "board-worthy". Escalate between meetings when an incident traced to an agent-authored change reaches customers or a regulator, when a loop drops a level after an incident, when an agent credential is revoked for cause, or when spend runs more than a pre-agreed margin over budget. The [agent incidents guide](/en/org/agent-incidents/) covers containment and the postmortem that feeds the next report. ## What should you not claim to the board? This section of the template is the one that protects you. Each claim below appears in vendor decks and all-hands slides, and each fails under a prepared question. | Do not claim | Why it fails | Report instead | | --- | --- | --- | | "X% of our code is written by AI" as proof of value | A share of lines is not a share of work or of value. DX's industry figure, a 51.9% average across more than 400 companies (June 2026), measures authorship, not outcome | Cost per accepted change and the incident trend | | "Pull requests are up, so productivity is up" | Faros AI's telemetry of 22,000 developers (April 2026) shows throughput and quality moving apart: PR merge rate per developer +16.2%, but incidents per pull request +242.7% and median time in review +441.5% | Accepted changes and incidents per 100 accepted changes, side by side | | "AI makes our developers N% faster" | No general measurement supports it. METR's 2026 follow-up has point estimates in AI's favour, but the intervals cross zero and METR says selection makes the results hard to interpret | Lead time for accepted changes in named loops, against a baseline from a [designed pilot](/en/org/pilot-design/) | | "We can cut headcount by a factor of three to five" | No third-party measurement supports any productivity multiplier | Where freed capacity went, per the [business case](/en/strategy/business-case/) | | "Anthropic merges over 80% AI-authored code, so we can too" | Anthropic's own figure ("more than 80% of the code we merge", as of May 2026, Anthropic Institute) is vendor-internal: it describes that company's harness and test suite, not an industry benchmark | Your own level per loop and the gate that holds it | | "DORA says AI improves delivery" | DORA's 2025 report pairs a positive relationship with throughput with a negative relationship with stability; quote both halves or neither | Your change failure rate split by provenance | | "Time saved is money saved" | Saved hours become cash only when capacity is redeployed or a cost is avoided | The redeployment decision, recorded with an owner | If an executive wants one number, give cost per accepted change with its four-quarter trend. It includes review, rework and incidents, so it cannot improve by generating more code that nobody can verify. ## How do you run the quarterly reporting cycle? 1. **Week −3: freeze definitions and export.** The engineering operations or platform owner runs the versioned queries for every metric and saves the raw exports, with query IDs, to a folder such as `board/2026-q3/`. Nothing is computed by hand. 2. **Week −3: loop owners set their levels.** Each owner answers the ladder questions for their loop and links the last green run of the gate that justifies the level. A level without a gate link is reported one rung lower. 3. **Week −2: draft with an agent.** Use the drafting prompt below in Claude Code, Codex or Cursor. The agent only arranges numbers that already exist in the exports and footnotes the source of each one. 4. **Week −2: attack the draft.** Run the sceptical-chair prompt below, then fix or declare every finding. This is the step that catches a flattering trend built on a changed definition. 5. **Week −1: verify and sign.** Finance reconciles the cost block with invoices and co-signs it. The CISO signs the security row, legal signs the IP and regulation rows, and the CTO signs the page. The signatures go in the header. 6. **Week −1: send the pre-read.** The chair gets the page and the appendix a week ahead. The meeting discusses the decisions in section 6, not the numbers. 7. **After the meeting: record decisions and targets.** The target levels per loop become next quarter's column, and the [transformation roadmap](/en/strategy/transformation-roadmap/) gates are updated to match. The drafting and attacking steps work identically in all three tools: open the agent in the folder that holds the exports and paste the prompt. For an unattended run, use `claude -p --permission-mode dontAsk "…" > board/2026-q3/draft.md` in Claude Code, or `codex exec -s read-only -o board/2026-q3/draft.md "…"` in Codex; both only read the exports and print the draft. In Cursor, paste the prompt into the agent with the folder open. **Copy-paste prompt: draft the board one-pager from the exports** ```text You are drafting the quarterly AI engineering board update. The folder board/2026-q3/ holds the raw exports (CSV and JSON), metric-definitions.md, and last quarter's report in board/2026-q2/report.md. Fill the one-page template in board/template.md. Rules: 1. Use only numbers present in the exports or last quarter's report. Do not estimate, extrapolate or round beyond one decimal place. 2. After every number, add a footnote naming the file, the query ID and the row it came from. 3. If a metric has no data, write "not measured" and list it under "Gaps" at the end. Never fill a gap. 4. Split every quality metric by the provenance label (agent / human). If the label is missing on more than 10% of changes, say so in the headline. 5. Compare metric-definitions.md with last quarter's copy. If any definition changed, flag the affected trend as "not comparable". 6. Do not write any productivity multiplier, headcount saving, or "% of code written by AI" as a measure of value. 7. Keep the page to one printed page; move detail to an appendix. Output the filled template, then the footnotes, then the gaps list. ``` **Copy-paste prompt: attack the draft as a sceptical audit committee chair** ```text Act as the chair of our audit committee: a former CFO, sceptical of technology claims, who reads the pre-read the night before. Review board/2026-q3/draft.md against the exports in board/2026-q3/. List every problem you find, most serious first: - a number that does not match its footnoted source, or has no source; - a trend that crosses a definition change; - a throughput number reported without the matching quality number; - a level above L2 without a linked gate run; - a risk row with no leading indicator, no owner or evidence older than this quarter; - any sentence implying a productivity multiplier, a headcount saving, or causation the data cannot show. For each problem, write the question you would ask in the meeting and the fix. Do not rewrite the report. ``` ## How do you know the report is right before it reaches the board? Nobody should have to re-read the page line by line to trust it. Build the checks into the cycle so each one produces evidence someone signs: - **Reproducibility.** Every number has a query ID in a versioned repository, and a second person reruns the queries and gets the same values. A mismatch blocks the pack. - **Reconciliation.** Finance reconciles total spend with invoices within an agreed tolerance before co-signing. - **Sampled audit of "accepted".** Draw 10 accepted changes at random and confirm each met its acceptance criteria and was not reverted. One failure means the definition or the data is wrong, and the trend line waits. - **Provenance coverage.** Report the share of changes carrying a provenance label. Below your threshold, the agent/human split is marked "not comparable". - **Gate evidence.** Each level above L2 links to the last green run of its gate, dated within the quarter. - **Named sign-offs.** CTO, CFO, CISO and general counsel sign their blocks, and the signatures sit in the header, not in an email thread. The agent drafts and attacks; people sign, the same division this site asks engineers to apply to code. ## When AI board reporting goes wrong **The metric becomes the target.** Teams split work into smaller changes to lower cost per accepted change. Recovery: report median change size beside it; DX measured median pull-request size rising from 44 to 72 lines between July 2025 and June 2026 (June 2026), so a sudden fall in your data is worth a question. Change the unit to accepted outcomes if splitting persists. **The numbers do not reconcile with finance.** Vendor dashboards and invoices disagree, usually because of seats bought but unused, metered overage, or a second tool on expenses. Recovery: take spend from invoices only, report the dashboard gap as a note, and move the fix into [cost governance](/en/org/cost-governance/). **Attribution breaks.** A move to Zero Data Retention switches off Claude Code's contribution metrics, or a tool switch changes the labels. Recovery: your own provenance label in the pipeline, set by the [PR labelling guide](/en/cto-scorecard-guide/ai-pr-labeling/), so the series continues across vendors. **A loop drops a level.** After an incident, a loop falls from L4 to L3. Report it plainly: the drop is the control working, and hiding it costs more credibility than the incident. Link the postmortem and the new eval that came out of it. **The board asks for "the ROI percentage".** A single ROI figure invites the multipliers the "what not to claim" table rules out. Recovery: give cost per accepted change with its trend and point to the decision recorded in the [business case](/en/strategy/business-case/). **The report grows to 12 pages.** Recovery: a hard one-page limit, an appendix for everything else, and a rule that a new row replaces an old one. ## Where to go next with board reporting - Set the cost method behind the economics block: [the economics of agent-built software](/en/strategy/economics/). - Get the baseline that makes the first trend meaningful: [designing a pilot that proves something](/en/org/pilot-design/). - Settle the metric definitions across the organisation: [DORA, SPACE, DX Core 4 and AI measurement](/en/org/metrics-frameworks/). - Check the gates behind each target level: [the organisation-wide transformation roadmap](/en/strategy/transformation-roadmap/). - Find where your organisation sits today, with a risk register to start from: take the free [C-level scorecard](/en/cto-scorecard/). --- # Software built outside engineering URL: https://developertoolkit.ai/en/strategy/builders-beyond-engineering/ Description: How analysts, ops, and domain experts build internal tools with agents and app builders: where it pays off, how prototypes graduate, where governance stops. Software built outside engineering is the internal tools, reports, and prototypes that analysts, operations staff, and domain experts now build themselves with coding agents and app builders such as Lovable, Replit, v0, and Bolt. It pays off for disposable prototypes and single-team tools on approved data. Anything with real users, sensitive data, or external effects moves into engineering's verified delivery. This page is for executives and CTOs. The situation it answers: your head of operations demos a scheduling app on Monday that she built over the weekend in an app builder. It already has 30 colleagues using it, it reads the HR system through her personal API token, and nobody in engineering has seen the code. You want more of this energy, not less. You also need to know what happens when she goes on holiday and the app breaks. ## What you can adopt for builder-made software - A **payoff table** that tells you which kinds of builder-made software are worth encouraging and which belong to engineering from day one. - A **four-lane model** (disposable, controlled pilot, production, forbidden) that matches the [prototype policy](/en/cto-scorecard-guide/team-vibe-coding-policy/) in the CTO scorecard. - An **intake record** you can paste into a form, so every builder-made tool has an owner, a lane, and an expiry on the day it is created. - A **graduation procedure** with copy-paste prompts, which turns a working prototype into acceptance criteria that engineering can verify without reading every line. - The **red lines** that no lane crosses, the **metrics** that show whether the programme works, and the questions to put to your CTO or platform lead. This page builds on [the economics of agent-built software](/en/strategy/economics/); read it first for how to count the cost of graduation. ## Who builds software outside engineering now? Three groups build software outside engineering, and they need different controls. | Group | Typical tools | What they build | What they usually lack | | --- | --- | --- | --- | | **App builders** | Lovable, Replit, v0, Bolt: a chat prompt produces a running web app | Dashboards, intake forms, small workflow apps, clickable prototypes | A repository your company owns, tests, a named maintainer | | **Agent users outside engineering** | The Claude Code desktop app, the Codex desktop app, Cursor | Scripts, data pipelines, integrations, spreadsheet replacements | Sandboxing, secret handling, code review | | **Analysts** | The same agents plus SQL and notebooks | Reports, one-off analyses, recurring data jobs | Version control, reproducibility, data classification | Bolt's own repository describes the category's loop plainly: "Prompt, run, edit, and deploy full-stack web applications" (`stackblitz/bolt.new`, read 2026-09-26). Plans, prices, and enterprise controls for Lovable, Replit, v0, and Bolt change often; confirm them in each vendor's current documentation (this page does not state them, as of 2026-09-26) before you sanction one, and use the questions in [how to choose a sanctioned builder](#how-do-you-choose-a-sanctioned-app-builder) below. Andrej Karpathy draws the line this page builds on: "Vibe coding raises the floor. Agentic engineering is about extrapolating the ceiling" (his published transcript of his Sequoia Ascent talk, 2026-04-30). Earlier in the same passage he says: "You are still responsible for your software, just as before." Builders outside engineering raise the floor. Your job is to decide where the floor ends and who owns what sits above it. ## Where does software built outside engineering pay off? Builder-made software pays off where the person who understands the problem can also judge whether the result is right, and where a mistake stays small. The table ranks the common cases. | Use case | Payoff | Why | Default lane | | --- | --- | --- | --- | | Clickable prototype to settle what a feature should do | High | Replaces weeks of requirement documents with something users react to; the prototype becomes the spec | Disposable | | One-off analysis or report on approved data | High | The analyst can check the numbers against a known source | Disposable | | Single-team internal tool on non-sensitive data | Medium to high | Removes a queue for engineering; the team can tell when it is wrong | Controlled pilot | | Recurring data job feeding other teams' decisions | Medium | Useful, but others now depend on it and cannot see its errors | Controlled pilot, graduation review at the first new consumer | | Automation that writes to a system of record (CRM, ERP, HR) | Low without engineering | A silent error corrupts shared data; rollback is hard | Production from day one | | Anything customer-facing, with logins, payments, or personal data | Negative without engineering | Security, privacy, and liability exceed what the builder can verify | Production or forbidden | Two things make the payoff real. First, the builder must be able to verify the output against something they already trust: a known total, a manual process, a colleague's judgment. Second, the tool must be cheap to throw away. When either condition fails, the value moves to engineering, because only engineering has the tests, review, and rollback that make the software trustworthy. Google Cloud's announcement of the 2025 DORA report (2025-09-23) puts the organisational side this way: "AI doesn't fix a team; it amplifies what's already there." Builders outside engineering amplify whatever data discipline and ownership habits your company already has. ## Which lane does each builder-made tool belong in? Use the four lanes from the [prototype policy](/en/cto-scorecard-guide/team-vibe-coding-policy/). The lane follows exposure and risk, never the tool that produced the code. The table adds who owns each lane and what the builder may do in it. | Lane | Boundary | Builder may | Owner | Evidence before anyone relies on it | | --- | --- | --- | --- | --- | | **Disposable exploration** | Synthetic or approved data, no external users, no writes to shared systems | Build and share a demo; expiry of 30 days or less | The builder | Purpose, owner, and expiry in the inventory | | **Controlled pilot** | Named users inside one team, approved data, reversible effects | Run it for the team on the sanctioned platform | The builder, plus a named engineering sponsor | Acceptance criteria, a test run against them, monitoring, a rollback note | | **Production system** | Other teams, customers, persistent data, or material effects | Contribute the prototype and the acceptance criteria | An engineering team | The full [lifecycle](/en/method/lifecycle/) loop: spec, tests, review, [evidence bundle](/en/workflows/ship/evidence-bundle/), named approver | | **Forbidden path** | Secrets, unapproved personal or regulated data, payments, destructive access | Nothing; stop and use the approved route | Security | Not applicable | The 30-day expiry is a starting value, not a benchmark. Pick yours and write it down. What matters is that every disposable tool has one, so an expired tool is either renewed deliberately or deleted. ## What should the intake record capture? Record every builder-made tool when it is created, not when it breaks. The template below fits in any form tool. Keep it to one screen, or builders will skip it. ```text Tool name: Builder (name, team): Engineering sponsor (required for controlled pilot and above): Purpose, in one sentence: Lane: disposable | controlled pilot | production | forbidden Users today (count, teams): Data it reads (system, classification): Data it writes (system, or "none"): External effects (email, payments, API calls to third parties, or "none"): Credentials used (service account name, never a personal token): Where the code lives (company repository URL, or builder workspace URL): Expiry date: How the builder checks it is right (the trusted source or manual check): ``` The last field is the most important one. A builder who cannot say how they check the output is running a tool nobody verifies. ## How does a prototype graduate to production? A prototype graduates when a trigger fires: new users outside the team, sensitive data, writes to shared systems, or someone else depending on it. Graduation turns the prototype into a specification that engineering implements and verifies, instead of a codebase engineering inherits blind. 1. **Freeze the behaviour.** The builder records what the tool does today: screens, inputs, outputs, and three to five real examples with the expected result. The examples become test cases. 2. **Extract acceptance criteria from the prototype.** An engineer, or the builder with an engineer reviewing, runs an agent over the prototype's code and the builder's examples to draft [acceptance criteria](/en/workflows/plan/acceptance-criteria/). The builder confirms each criterion matches what users actually need. 3. **Classify the risk.** Place the future system in the [four risk tiers](/en/org/governance-autonomy/) your engineering organisation uses. Authentication, payments, or production data make it Tier 3 no matter how small the code is. 4. **Decide: harden in place or rebuild.** Use the decision table below. Neither answer is the default. 5. **Implement under engineering's gates.** The owning team works in a company repository with CI, tests derived from the acceptance criteria, security review, and an evidence bundle. The builder signs off that the behaviour matches; the engineering owner signs off that it is safe to run. 6. **Retire the prototype.** Revoke its credentials, delete its data copies, take down its domain, and cancel its builder workspace. Record the retirement in the inventory. | Signal | Harden in place | Rebuild in the production stack | | --- | --- | --- | | Code can move to a company repository | Yes | No, or only as an export nobody can run | | Stack matches something your platform already runs | Yes | No | | Data model is sound for the real volume and access rules | Yes | No, or unknown | | Security review finds isolated issues | Yes | Systemic issues (auth, secrets in code, no access control) | | The prototype's main value is the behaviour it demonstrates | Either | Usually rebuild: keep the spec, drop the code | Rebuilding does not waste the prototype. The expensive part of most internal tools is discovering what they should do, and the prototype has already paid for that. **Copy-paste prompt: turn a prototype into acceptance criteria** ```text You are reviewing a prototype built outside engineering. Read the code in this repository and the file examples.md, which lists real inputs and the results the builder expects. 1. Describe what the tool does in plain language, screen by screen. 2. Write acceptance criteria in Given/When/Then form, one per behaviour. Every example in examples.md must be covered by at least one criterion. 3. List behaviour the code implements that no example covers, and mark each one "confirm with builder". 4. List every external system the code reads or writes, with the credential it uses and where that credential is stored. Do not modify any file. Output one markdown document with four sections. ``` **Copy-paste prompt: graduation review of a builder-made app** ```text Assess this application for graduation from a prototype to a production system. Check and cite file and line for each finding: - Authentication and authorisation: who can reach each page and API route. - Secrets: any key, token, or password in code, config, or client bundles. - Data: which personal or confidential fields it stores, where, and for how long. - External effects: emails, payments, writes to other systems. - Dependencies: every package, flagged if it is unmaintained or if you cannot confirm it exists on the public registry. - Tests: what is covered, and what is not. Classify the result as: keep disposable, harden in place, or rebuild. Give the three findings that most affect that decision first. Do not change any code. ``` The dependency check in the second prompt matters more for builder-made code than for engineering code. Researchers found that 19.7% of 2.23 million code samples from 16 models referenced at least one package that does not exist (USENIX Security 2025, Spracklen et al., as reported second-hand by Aikido and the Cloud Security Alliance). A builder who has never run a package manager cannot spot one. ## How does engineering run the graduation review in each tool? The prompts above work in any of the three agents. What differs is how you keep the review read-only and where the second opinion comes from. **Claude Code** Start the session in plan mode so the agent reads and reports without editing: `claude --permission-mode plan`. Paste the graduation prompt. Then run `/security-review` for a dedicated security pass, and `/code-review` on the pull request once the hardened version exists. For a builder who should only read and analyse, `claude --restricted` removes the built-in tools that run commands or code, as well as WebFetch, and ignores user, project, and local settings files, while [managed settings](/en/org/managed-policy/) still apply (CLI flag, checked on v2.1.283). One default matters for builders outside engineering: from v2.1.283 (both release channels on 2026-10-12), interactive terminal and VS Code sessions start in auto mode on supported models, where a classifier approves actions instead of the person (`claude -p` and the Agent SDK start in Manual, except on third-party providers or with telemetry off, where they start in auto mode from v2.1.285; sessions on unsupported models such as Haiku 4.5 start in Manual too). That suits an engineer who can read a shell command. For builders who cannot, decide deliberately, and switch it off for the whole organisation with `permissions.disableAutoMode: "disable"` in managed settings. **Codex** Import the prototype into a company repository first. Run the graduation prompt in the Codex CLI or desktop app with a read-only permission profile: `codex -c default_permissions=":read-only"` (permission profiles are beta, checked on 0.157.1). After hardening, `codex review --base main` reviews the branch non-interactively, which fits as a CI step. Admins can pin permission profiles and restrict MCP servers for everyone through `requirements.toml`. **Cursor** Open the imported repository and start in **Plan Mode**, which "creates detailed implementation plans before writing any code" (Cursor docs, checked 2026-08-28), then paste the graduation prompt. Once the hardened version is a pull request, **Bugbot** reviews it for bugs, security issues, and code-quality problems. Cursor's current admin controls for non-engineering seats could not be verified from cursor.com on 2026-09-26; confirm them with your account team before you hand Cursor to builders. ## Where does governance draw the line? Some lines hold in every lane, for every builder, in every tool. Write them into your [AI usage policy](/en/org/usage-policy/) and enforce them technically, because a policy document does not stop a pasted token. | Red line | Why | How to enforce it | | --- | --- | --- | | No personal API tokens or production credentials in builder tools | A personal token gives the app its builder's full access, and it leaves with them | Service accounts with narrow scopes, issued through the [agent identity and secrets](/en/org/agent-identity-and-secrets/) process | | No personal or regulated data without a data-protection decision | The builder vendor becomes a processor; retention and location matter | Data classification plus the vendor review in [privacy and data handling](/en/org/privacy-data-handling/) | | No payments, no authentication for external users | These are the failures that become incidents and liability | Forbidden lane; engineering builds them | | No writes to a system of record without an engineering owner | Silent data corruption spreads to every downstream team | API gateways that allow writes only from registered service accounts | | No public exposure without review | A public URL turns an internal toy into an attack surface | SSO in front of every sanctioned builder; public publishing disabled by default | | No tool without an owner and an expiry | Orphaned tools become unowned production | Inventory with automatic expiry reminders | Regulated firms add their own lines. If a builder-made tool touches decisions about people, such as hiring, credit, or access to services, check it against [the EU AI Act](/en/org/eu-ai-act/) before it leaves the disposable lane. ## How do you choose a sanctioned app builder? Offer one sanctioned route per group, so people stop using whatever they found. DORA's guidance on AI policy explains why a sanctioned route beats a ban: "Ambiguity creates risk. A clear policy provides the psychological safety developers need to experiment effectively" (Google Cloud blog, 2025-12-10). The same holds for builders outside engineering. Ask every vendor these questions, and put the answers in the [procurement record](/en/org/procurement/): 1. Can we enforce company SSO and disable personal accounts on the company domain? 2. Can the code sync to a repository our organisation owns, and does the export run outside the vendor? 3. Can we disable public publishing, or require SSO in front of every published app? 4. Where are app data and prompts stored, for how long, and are they used for training? 5. Can we set a spending cap per workspace and see usage per user? 6. Can admins list every app in the organisation, with its owner and last activity? A builder that fails questions 1 or 2 can still serve the disposable lane on synthetic data. It cannot host a controlled pilot. For the agent-user and analyst groups, the sanctioned route is usually the same agent engineering already runs, set up with tighter managed settings and without production credentials. That keeps one set of controls and lets graduation happen inside the same toolchain. ## How do you know the programme is working? Measure the programme, not the builders. Each metric has an owner and a definition you can adopt as written. | Metric | Definition | Owner | Healthy direction | | --- | --- | --- | --- | | Inventory coverage | Registered builder-made tools ÷ tools found by discovery (SSO logs, builder admin consoles, expense reports), per quarter | CTO's platform or security lead | Rising towards all of them | | Owner-and-expiry rate | Registered tools with a named owner and a future expiry date ÷ all registered tools | Platform lead | Near all | | Graduation lead time | Days from a graduation trigger firing to the tool running under engineering's gates, or being retired | Engineering sponsor | Stable or falling | | Unowned production | Tools with users outside the builder's team and no engineering owner | CTO | Zero | | Incidents from builder-made tools | Security or data incidents whose root cause is a builder-made tool, per quarter | Security | Zero, and every one reviewed | | Retirement completeness | Retired tools whose credentials, data, and domains were verified removed ÷ all retired tools | Platform lead | All | Do not report "hours saved by citizen developers" to the board. Nobody measures the counterfactual, and time saved is not cash saved, as [the economics of agent-built software](/en/strategy/economics/) explains. Report the metrics above, plus the named tools that graduated and what they replaced. ## What goes wrong with software built outside engineering? **A disposable tool becomes production without anyone deciding it.** It gains users, then data, then a dependency. Recovery: run discovery monthly, and treat any tool with users outside its team as a graduation trigger that fires automatically. **A personal token powers a shared app.** The builder leaves, the token is revoked, and a team's workflow stops overnight. Recovery: rotate to a service account before anything else, then register the tool and assign a sponsor. **Engineering becomes the gate everyone routes around.** If graduation takes a quarter, builders stop registering tools. Recovery: publish the graduation lead time, staff a small rotation of engineering sponsors, and make the disposable lane need no approval at all. **Engineering rebuilds and loses what the prototype knew.** The rebuilt tool is cleaner and wrong. Recovery: make the builder's examples the acceptance tests, and require the builder's sign-off on behaviour before release. **Builder sprawl.** Five teams use five builders, each with its own data copy. Recovery: sanction one builder per group, and migrate the rest at their next graduation or expiry. **Cost surprises.** Usage-based builder plans grow with enthusiasm, not value. Recovery: a spending cap per workspace, reviewed alongside [cost governance](/en/org/cost-governance/) for engineering's own agents. ## Questions to put to your CTO or platform lead about builder-made tools A CTO can use the same list as a self-check before the board asks. - How many builder-made tools do we run today, and how do we know that number is complete? - Which of them have users outside the builder's team, and who owns those? - Which builder tools are sanctioned, and do they meet the six procurement questions? - How long does graduation take, and how many tools graduated or retired last quarter? - Which builder-made tools use personal tokens or touch personal data? - What would we see first if one of them caused an incident? ## Where to go next with builder-made software - [Prototype policy](/en/cto-scorecard-guide/team-vibe-coding-policy/) — The scored CTO policy: lanes, graduation triggers, and acceptance evidence. - [Product management when build time collapses](/en/strategy/product-management/) — How product teams author intent and acceptance criteria, and keep prototypes out of production. - [Governance and autonomy](/en/org/governance-autonomy/) — The four risk tiers and the controls each one requires. - [How teams, roles, and headcount change shape](/en/strategy/org-design/) — Where builders, verifiers, and engineering sponsors sit in the new org design. When a builder-made tool graduates, the owning team follows [acceptance criteria](/en/workflows/plan/acceptance-criteria/) and the [evidence bundle](/en/workflows/ship/evidence-bundle/) like any other change. --- # Building the business case for agentic engineering URL: https://developertoolkit.ai/en/strategy/business-case/ Description: A decision-memo template for funding agentic engineering: four options, a measured baseline, full cost, a break-even figure, stop gates and risks. A business case for agentic engineering is a decision memo that compares four options (do nothing, seats only, a shared harness, a factory pilot) against a measured baseline, with full cost, a break-even figure instead of a vendor multiplier, and stop gates written before money moves. It asks for a bounded experiment, not a productivity forecast. Your CTO wants budget for coding agents. The vendor deck promises a multiplier; your CFO has read that METR's 2025 randomised trial found experienced developers took 19% longer with AI. Neither number describes your codebase, review capacity or incident rate. This page gives the executive who approves the spend, and the CTO who defends it, a memo that survives that CFO. ## What this business-case template gives you - A one-page decision memo in Markdown, with its options table, full-cost checklist, stop gates and risk register - A break-even calculation that replaces "3x faster" with "this pays back if accepted changes rise by N% at constant headcount and quality" - Two prompts: one pulls the baseline from your repository, one red-teams the memo before your CFO does ## Why a multiplier-based business case fails with a CFO A multiplier loses the room once someone asks for its source, and third-party evidence points both ways: | Source | What it measured | What it found | | --- | --- | --- | | METR, randomised trial, 2025-07-10 | 16 experienced open-source developers, 246 issues, early-2025 tools | Tasks took 19% longer with AI (interval +2% to +39%), yet the developers believed AI had sped them up by 20%. | | METR, follow-up, 2026-02-24 | The same design, late-2025 tools | Point estimates favour AI (about 18% and 4% less time in two cohorts), but both intervals cross zero; METR calls the data "an unreliable signal", hard to interpret because of selection. | | DORA, 2025 report, 2025-09-23 | Survey of software professionals | "A positive relationship between AI adoption on both software delivery throughput and product performance", and "a negative relationship with software delivery stability". | | Faros AI, April 2026 | Vendor telemetry: two years, 22,000 developers, 4,000+ teams | Epics per developer +66.2% and tasks per developer +33.7%, while bugs per developer rose 54%, incidents per pull request 242.7% and median time in review 441.5%. | Three conclusions follow: 1. **Self-reports are not a baseline.** METR's developers misjudged their own speed, so "hours saved" surveys cannot fund a budget line. 2. **Throughput gains arrive with a quality bill.** DORA and Faros both show more output with less stability. DORA names the mechanism: "Without robust control systems, like strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability." 3. **The honest ask is for an experiment.** No study measures your codebase, and vendor or consultancy multipliers without a stated method measure it even less. So the memo asks for a bounded trial with a written decision rule and states the cost of being wrong. The [state of agentic engineering](/en/ladder/state-of-agentic-engineering-2026/) page collects the dated evidence. ## Which four options should the memo compare? Compare the same four options every time, so "do nothing" is costed as seriously as the ask. Levels refer to the [autonomy ladder](/en/ladder/), which rates each delivery loop, not the company. | Option | What you buy | Ladder reach | What it can prove | Main risk | Reversible? | | --- | --- | --- | --- | --- | --- | | **A. Do nothing** | No sanctioned tools | L0–L2, unmanaged | Nothing | Shadow use with no data controls and no measurement | Yes | | **B. Seats only** | Licences for every engineer | L2–L3 per person | Adoption and satisfaction; weak evidence on outcomes | More diffs than reviewers can read; the Faros pattern | Yes, at renewal | | **C. Shared harness** | Seats plus shared rules, skills, hooks, CI gates and review agents, owned by a named person | L3, with a path to L4 on chosen loops | Cost per accepted change and quality against a matched control team | Harness work competes with feature work | Yes; the harness is files in the repository | | **D. Factory pilot** | One narrow loop run unattended against an oracle (such as dependency upgrades) | L4 on one loop | Whether a loop can run with evidence instead of line-by-line review | Runaway usage; a weak oracle passes bad changes | Yes, if the loop is switched off by config | "Do nothing" is not the zero-risk option. DORA's 2025 report found that 90% of survey respondents use AI at work. Without a sanctioned tool, engineers use personal accounts with no data terms, audit trail or measurement. Recommend **C** and **D** together: C for feature teams, D for one loop whose oracle is already strong. B alone adds generation without verification: the throughput-up, stability-down pattern. ## How to build the business case, step by step 1. **Write the decision question and the decision date.** For example: "Do we fund option C for 12 teams in Q1, based on an eight-week, two-team pilot ending on 2026-12-18?" A memo without a date becomes a standing budget line. 2. **Measure the baseline from systems, not surveys.** For the 8–12 weeks before any change, take accepted changes per month (merged and not reverted within 30 days), lead time, change-fail rate, median review time, incidents and rework, with the definitions frozen from [the metrics frameworks guide](/en/org/metrics-frameworks/). Compute today's cost per accepted change ([the economics guide](/en/strategy/economics/)) and the fully loaded monthly cost of the delivery organisation. 3. **Cost each option in full** with the checklist in the next section. Finance supplies the loaded cost per engineer; engineering supplies usage, CI and review hours. 4. **State the case as a break-even figure**, as in the worked example below. 5. **Write the stop gates and the decision rule** before the pilot starts. [Pilot design](/en/org/pilot-design/) covers cohorts, confounders and sample size. 6. **Fill in the risk register:** an owner, an early signal and a funded control for every risk. 7. **Get three signatures:** the CFO for the baseline and cost model, the CTO for the gates and oracle, and a named analyst outside the pilot team for the result. ## What goes into the full cost of each option? Licences are the smallest line that moves; usage, review time and the quality bill decide the case. | Cost line | Where the number comes from | Owner | Options it applies to | | --- | --- | --- | --- | | Seats and plan fees | Vendor quote; list prices on [the pricing analysis](/en/comparison/pricing-analysis/) | Procurement | B, C, D | | Metered usage and overage | Tool analytics and exports (see the tabs below); model rates on [the models hub](/en/ecosystem/models/) | Engineering | B, C, D | | CI compute | CI billing, before and after | Platform | C, D | | Harness ownership | Hours of the named owner for rules, skills, hooks and gates | CTO | C, D | | Oracle build | Engineering hours to bring tests or evals up to the bar the loop needs | Tech lead | D | | Review time | Median review time × accepted changes, costed at the loaded rate | Engineering | B, C, D | | Rework | Changes reverted or rewritten within 30 days | Engineering | All | | Incidents | Incidents attributed to changes × average cost of an incident | SRE | All | | Training and the adoption dip | Hours in training; a planned period of lower throughput | Engineering | B, C, D | | Security, legal and procurement review | One-off hours plus any new tooling | CISO, legal | B, C, D | For metered usage, Anthropic's Claude Code documentation reports that across enterprise deployments "the average cost is around $13 per developer per active day and $150-250 per developer per month, with costs remaining below $30 per active day for 90% of users" (checked 2026-09-26). Treat it as a vendor planning input until pilot data replaces it. Unattended loops cost more: DoltHub's Tim Sehn reported about $100 in Claude tokens for one hour of a multi-agent orchestrator (2026-01-15) and $3,000 for a week (2026-03-24). One user's anecdote, but it is why option D needs a hard budget cap. Budget the adoption dip: DORA's ROI report describes a J-curve, an initial dip before returns, according to InfoQ's May 2026 coverage (secondary source; the report itself was not reviewed). Write its expected length into the memo as an assumption, for example four to six weeks, and test it. ### Where each tool reports the cost data you need Cost per accepted change needs a cost record from every agent run, and each tool keeps one differently. **Claude Code** The Team and Enterprise analytics dashboard shows adoption and PR attribution but not per-user cost; for per-user, per-model dollars export the OpenTelemetry metric `claude_code.cost.usage` (see [cost governance](/en/org/cost-governance/)). For option D, run each unattended task headless with a budget cap. `claude -p` starts in Manual permission mode (auto mode on third-party providers or with telemetry off, from v2.1.285), so pre-approve every tool the loop needs, or the run records a cost without changing anything: ```bash # Terminal or CI: one unattended task, capped at $5, cost recorded with the result claude -p "Upgrade the date-fns dependency to the latest minor version, fix any failing tests, and stop when npm test passes" \ --permission-mode acceptEdits --allowedTools "Bash(npm install *)" "Bash(npm test *)" \ --output-format json --max-budget-usd 5 > run.json jq '{cost_usd: .total_cost_usd, is_error: .is_error}' run.json ``` `total_cost_usd` is a client-side estimate at list prices; reconcile it monthly against the invoice. **Codex** Codex plans meter usage in five-hour and weekly allowance windows, so a seats-only pilot shows headroom, not cost. For option D, run tasks with `codex exec --json`: each `turn.completed` event carries a `usage` object with `input_tokens`, `cached_input_tokens`, `cache_write_input_tokens`, `output_tokens` and `reasoning_output_tokens`; multiply by the rates on [the models hub](/en/ecosystem/models/). The upgrade needs the package registry, and the `workspace-write` sandbox blocks network access by default, so this run turns it on: ```bash # Terminal or CI: one unattended task, token usage per turn written to a log codex exec --json --sandbox workspace-write -c sandbox_workspace_write.network_access=true \ "Upgrade the date-fns dependency to the latest minor version, fix any failing tests, and stop when npm test passes" \ | jq -c 'select(.type == "turn.completed") | .usage' >> usage.jsonl ``` Field names and the config key were checked in codex-cli 0.157.1, which has no per-run dollar cap; bound the run with a CI job timeout and the spend-ceiling gate instead. `--sandbox` is the legacy control, used here because network access under a permission profile was not verified in 0.157.1; do not combine it with a profile, OpenAI's preferred replacement ([permissions and sandboxing](/en/workflows/harness/permissions-and-sandboxing/)). **Cursor** Cursor's CLI runs the same pilot tasks headless in print mode (`-p`). Its cost fields, usage exports and plan billing are not verified here as of 2026-09-26, so before the pilot ask Cursor for a per-user, per-model usage export separating included usage from overage. ## How much improvement does each option need to pay back? Replace the multiplier with a break-even figure: > break-even rise in accepted changes = added monthly cost of the option ÷ fully loaded monthly cost of the delivery organisation That is how much accepted changes must rise, at the same headcount and with the quality gates holding, to cover the option. The example below uses illustrative inputs for a 40-engineer organisation; replace each with your finance team's figures. | Input (illustrative) | Value | | --- | --- | | Fully loaded cost per engineer per month | $15,000 (about $94 an hour at 160 hours) | | Monthly cost of the delivery organisation | 40 × $15,000 = $600,000 | | Option C: seats plus usage, costed at the top of Anthropic's reported $150–250 monthly usage range | 40 × $250 = $10,000 | | Option C: harness owner, half an engineer | $7,500 | | Option C: extra CI compute | $1,500 | | One-off: training, 8 hours per engineer | 40 × 8 × $94 ≈ $30,000 | | One-off: security, legal and procurement review | $15,000 | | One-off: adoption dip, assumed four weeks (the low end of the range above) at 10% lower throughput | 0.10 × $600,000 = $60,000 | | One-offs amortised over 12 months | $105,000 ÷ 12 = $8,750 | | **Option C: added monthly cost** | **$27,750** | | **Break-even rise in accepted changes** | **$27,750 ÷ $600,000 ≈ 4.6%** | If you buy seats, replace the usage line with the quoted seat price from [the pricing analysis](/en/comparison/pricing-analysis/) plus expected overage. Review time has no line because reviewers are already in the $600,000; extra review load shows up as fewer accepted changes. The bar looks low, and that is the point for a CFO: the risk is the quality bill, not the tool cost. Rework and incidents rising as Faros measured would erase a 5% gain unnoticed, so the stop gates below are quality gates first. Write the outcome as three scenarios, not one number: - **Null:** no measurable rise. The organisation spent the stated pilot cost and learned where its verification is weak. - **Below break-even:** continue only if the quality gates held and the trend is rising. - **Case met:** above break-even with quality gates held. Scale to the next cohort, not to everyone. Time saved is not cash saved. The memo says where freed capacity goes (backlog shipped earlier, a hire not made, a contract ended), or the saving never reaches the P&L. ## Stop gates: write them before the money moves The gates turn a budget request into an experiment. Adjust the thresholds to your baseline; keep the structure. | Gate | Metric | Starting threshold | Checked | Action if breached | | --- | --- | --- | --- | --- | | Stability | Change-fail rate in the pilot cohort | No more than 2 points above baseline | Weekly | Two consecutive breaches stop the pilot | | Rework | Accepted changes reverted or rewritten within 30 days | No higher than baseline | Every two weeks | Pause new loops; find the missing test | | Review load | Median review time per accepted change | No more than 20% above baseline | Weekly | Cap pull request size; add a review agent before adding seats | | Protected changes | Unreviewed merges touching auth, payments, schema or migrations | Zero | Continuous, enforced in CI | Stop the loop that produced it | | Cost | Cost per accepted change (all six terms in [the economics guide](/en/strategy/economics/)) | No higher than baseline by week 8 | Monthly | Extend once for four weeks, then stop | | Spend ceiling | Usage spend per loop per week | A fixed dollar cap per loop | Continuous, by budget flag or gateway | The run stops; the operator reviews | The decision rule, written into the memo before the pilot starts, reads like this: "Scale option C to the next six teams if, after eight weeks, accepted changes per engineer in the pilot cohort are at least 10% above the matched control cohort, cost per accepted change is no higher than the control's, and no quality gate was breached in two consecutive weekly checks. Stop if any stability or protected-change gate is breached twice. Otherwise extend once for four weeks, then stop." The 10% bar sits above the 4.6% break-even as a detection margin, so noise between two small cohorts cannot pass for a gain. Set it from your own weekly variance with the [pilot design guide](/en/org/pilot-design/). ## The risk register for an agentic engineering business case | Risk | Early signal | Control | Owner | | --- | --- | --- | --- | | Stability falls as change volume rises (DORA 2025; Faros 2026) | Change-fail rate and incidents per change trend up | Quality gates above; tests and fitness functions before autonomy | CTO | | Review becomes the bottleneck | Review queue age and pull request size grow | PR size budget; review agents; [review queue practices](/en/teams/review-queue/) | Tech lead | | Usage cost overruns | Weekly spend per loop above plan | Per-run budget caps; spend alerts; [cost governance](/en/org/cost-governance/) | Engineering finance | | Prompt injection or secrets exposure | Broad agent credentials; untrusted issue text in prompts | Scoped agent identities; [the agent threat model](/en/org/agent-threat-model/) | CISO | | IP and licence exposure | No policy on code provenance or vendor data terms | Vendor terms reviewed; [legal and IP checklist](/en/org/legal-and-ip/) | General counsel | | Vendor lock-in or price change | Workflows depend on one vendor's proprietary features | Portable `AGENTS.md`, skills and MCP; [lock-in and portability](/en/org/lock-in-and-portability/) | CTO | | Skills erode and juniors stop learning | Seniors cannot explain merged changes | Deliberate practice and review rotation; Anthropic's 2025-12-02 internal study calls it the "paradox of supervision" | Engineering managers | | The measurement is gamed or confounded | "Accepted change" redefined; volunteers only | Definitions frozen in the memo; matched control cohort; analyst outside the team | CFO | ## How the business case itself is verified The memo is checked like code, against evidence the pilot team did not produce alone. - **The baseline** comes from version control, CI and the incident system, signed by the CFO first. - **Every pilot change** carries its [evidence bundle](/en/workflows/ship/evidence-bundle/): tests, review record, run cost. - **The result** is computed by an outside analyst against a matched control cohort over the same weeks. - **The decision** follows the pre-written rule; overrides are recorded with a reason. **Copy-paste prompt: pull the baseline from your repository** ```text You are helping me build a baseline for a business case. Work read-only: do not edit, commit or push anything. In this repository, for the 12 weeks ending today: 1. Use git log on the main branch to count merged changes per week. 2. Find reverts: commits whose message starts with "Revert" or that reference a reverted commit hash. Report the share of merged changes reverted within 30 days. 3. If the GitHub CLI (gh) is authenticated, list merged pull requests with gh pr list --state merged --limit 500 --json number,createdAt,mergedAt,additions,deletions and compute the median hours from creation to merge and the median changed lines per pull request. 4. Output one table with one row per week and these columns: merged changes, reverts, median hours to merge, median pull request size. 5. List every assumption you made and every number you could not compute, with the reason. ``` **Copy-paste prompt: red-team the memo as a sceptical CFO** ```text Read the attached business-case memo. Act as a CFO who has read METR's 2025 randomised trial, DORA's 2025 report and Faros AI's April 2026 telemetry report, and who approves nothing without a baseline. List, in order of severity: 1. Every number that has no source, or whose source is the vendor selling the tool. 2. Every cost line that is missing, especially review time, rework, incidents, CI and the adoption dip. 3. Every claim that treats time saved as cash saved without saying where the freed capacity goes. 4. Every stop gate that is missing a threshold, an owner or an action. 5. The three questions you would ask in the budget meeting. Do not rewrite the memo. Quote the sentence each finding refers to. ``` ## The decision-memo template Copy this into your document system: one page plus the risk register. ```markdown # Decision memo: