AI tooling ROI — the evidence behind a full-score answer

A full-score answer to CTO Scorecard Q23 is an evidence pack, not a percentage: cost per accepted change for a baseline and a current period, matched cohorts with confounders recorded, lead time and quality guardrails for the same work, and uncertainty stated as ranges. The method itself lives on one page, the economics of agent-built software.

This page is for the CTO who answered Q23 (“Do you have numerical ROI from AI tooling?”) and wants to know what the top answer requires. The usual situation: finance has asked for the return on the AI budget, one team lead says review time dropped 30%, another sends anecdotes, and nobody can say what a shipped change costs today compared with last year. This page tells you which pieces you are missing and routes you to the page that builds each one. It does not repeat the method.

What each answer to Q23 is worth, and what it lacks

Q23 is one of the Strategy and ROI questions in the CTO Scorecard. Find your current answer, then fix the gap in its row. Each row is one step; skipping from the first row to the last usually produces a spreadsheet nobody trusts.

Your answer	Points	What is missing	Next step
“Feels faster”	0	Any number from a system of record	Take a baseline of merged PRs, reverts and tool spend for the last quarter, from the cost-per-accepted-change method
Anecdotes from team leads	1	A definition that two people compute the same way	Adopt the accepted-change definition and the six cost terms as written, and freeze them for the period
One or two metrics, such as “30% less review time”	2	Full cost, a quality guardrail, a comparison group and uncertainty	Add the missing cost terms, pair the metric with change failure rate from metrics frameworks, and run a matched-cohort pilot
Matched baseline and current cohorts covering accepted lead time, quality, outcome and total cost, with uncertainty and confounders recorded	3	Nothing for Q23; keep the evidence current each quarter	Turn it into a funding decision with the business case and report it with the board reporting kit

What goes into the Q23 evidence pack?

Use this as the acceptance checklist for a 3-point answer. Every line names the page that owns the method, so your organization keeps one ROI method instead of one per team.

• Unit cost, twice. Cost per accepted change for the baseline period and the current period: licenses, usage, CI and verification, review, rework and incidents, over PRs merged and not reverted within 30 days. Owner: economics.
• A comparison that controls for the obvious. A matched or concurrent control cohort, a confounder register and a sample size chosen before the period starts. Owner: pilot design.
• Flow and quality for the same work. Accepted lead time with median and p90, plus change failure rate and escaped defects as guardrails that a lower unit cost cannot average away. Owner: metrics frameworks.
• Outcome, not activity. Freed hours count as money only through redeployed roadmap capacity, an avoided cost or an earlier revenue date. Otherwise report them as capacity. Owner: economics.
• Uncertainty in the open. Ranges instead of point multipliers, sample sizes beside every sampled number, and the share of incidents with no traced change.
• A decision rule written first. The threshold for expand, continue, narrow or stop, signed before anyone reads the result. Owner: business case stop gates; the funded bets go on the AI tooling roadmap.

The pack is the same whichever agents your teams run. Claude Code, Codex and Cursor differ only in where the usage number comes from, and the economics page lists the export for each tool.

Copy-paste prompt: find your Q23 gap

I am answering CTO Scorecard Q23: "Do you have numerical ROI from AI tooling?"
The full-score answer is: matched baseline and current cohorts covering accepted
lead time, quality, outcome and total cost, with uncertainty and confounders recorded.

Read the attached ROI material (slides, spreadsheet, dashboards export). For each of
these six items, answer PRESENT, PARTIAL or MISSING and quote the cell or sentence:
1. cost per accepted change for a baseline and a current period, all six cost terms
   (licenses, usage, CI and verification, review, rework, incidents);
2. a matched or concurrent comparison cohort and a confounder register;
3. accepted lead time (median and p90) for the same work;
4. change failure rate or escaped defects as a guardrail;
5. value shown through redeployed capacity, avoided cost or revenue timing, not hours x salary;
6. ranges, sample sizes and a decision rule dated before the result.
End with the one item to fix first and why. Do not invent a number that is not in the material.

How is the Q23 answer verified?

Nobody should have to read the spreadsheet cell by cell to trust it. The answer holds when it survives four checks:

1. It reproduces. A second person re-runs the queries from source data (version control, CI, invoices, postmortems) with the frozen definitions and gets the same unit cost within rounding.
2. It reconciles. The license, usage and CI lines match what finance paid. A gap usually means an API key billed outside the tools’ own reports.
3. The guardrails held. A lower cost per accepted change with a rising change failure rate fails the check, whatever the headline says.
4. Two signatures. The finance partner signs the cost lines, the engineering lead signs the denominator and the hour samples, and you, as CTO, own the definitions and any change to them.

Copy-paste prompt: red-team the ROI claim before the board sees it

Act as a skeptical CFO who has read METR's 2025 randomized trial and its 2026 follow-up on AI developer tools.
Review the attached ROI summary and list every claim that would not survive the
question "how do you know?". For each: the claim, the evidence offered, what is
missing (baseline, comparison cohort, full cost, guardrail, sample size, dated
decision rule), and the weaker statement the evidence does support. Treat vendor
case studies as context, never as evidence of our own return.

What goes wrong when you answer Q23?

Every team has its own ROI spreadsheet. One multiplies hours saved by an hourly rate, another counts PRs. The numbers cannot be added up or compared. Recovery: retire the local methods and compute one cost per accepted change from the economics template for every team.

The baseline was taken after the rollout. Without a pre-change period, any improvement is a guess. Recovery: reconstruct the baseline from version-control, CI and invoice history for the quarter before rollout, and label it reconstructed.

One good metric is presented as ROI. “30% less review time” is a 2-point answer: it says nothing about cost, rework or incidents. Recovery: put it on the sheet as one input to the review line, next to the other five cost terms.

Hours saved become a savings figure. A survey multiplied by salary ends up in the board deck, and finance cannot find the money. Recovery: move it to a capacity line and name the redeployment, avoided cost or revenue date that would turn it into cash.

A vendor case study stands in for your own evidence. Another company’s result describes their codebase and their review process, not yours. Recovery: cite it as context and let your own pilot decide.

Where to go next after Q23

Q22 comes before this question and supplies the measurements; Q24 turns the result into funded bets.

The economics of agent-built software The site's one ROI method: cost per accepted change, a worked example and the spreadsheet template.

Designing a pilot that proves something Matched cohorts, a confounder register, sample size and a decision rule signed in advance.

Q22 · AI metrics panel The balanced measurements this question depends on.

Q24 · AI tooling roadmap Fund capability bets with graduation and stop gates.