AI tooling roadmap — the portfolio-item template for capability bets

Level 2 How-to Updated 2026-09-26

An AI tooling roadmap item is one funded capability bet: a delivery constraint in a named loop, a falsifiable hypothesis, a full-cost budget, quality guardrails, and a stop gate that decides graduate, continue, narrow or stop on a dated review. The items are the tooling layer of the organization’s transformation roadmap, not a second roadmap beside it.

This page is for the CTO or VP Engineering who answered CTO Scorecard Q24 (“Do you have a 6–12 month AI tooling roadmap?”) and wants the top answer. The usual situation: the roadmap slide reads “Q1 roll out Cursor, Q2 evaluate Codex cloud tasks, Q3 MCP for Jira”, the board asks what the first two bought, and nobody can say which line to stop. This page gives you the template that fixes each line; the plan the lines belong to is the transformation roadmap.

What you’ll walk away with from the tooling portfolio

A portfolio-item template in YAML that your platform team can commit today
A decision table for graduate, continue, narrow and stop, with what each decision changes
A retirement checklist, so a stopped bet leaves no live credentials behind
Three copy-paste prompts that draft an item, audit the portfolio and prepare a gate review
The Q24 gap table: which answer you gave and the one step to the next

How does the tooling portfolio fit the transformation roadmap?

The transformation roadmap moves loops up the autonomy ladder, one rung at a time, and records them in a loop register. A loop is one recurring kind of change in one codebase with one owner, such as UI bug fixes in the web app. A tooling bet exists only to help a registered loop pass its next gate. So every portfolio item names the loop it serves and the transition it unblocks, for example web-app/ui-bug-fixes, L3 → L4.

That rule removes most tool shopping by itself. “Evaluate a new agent” with no loop and no gate is not a portfolio item; it is curiosity, and it belongs in a sandbox with a time box, under the exception process in the tooling policy.

Artifact	Answers	Owner	Lives in
Transformation roadmap	Which stage is the organization in, and which loops climb next?	CTO, executive sponsor	Transformation roadmap
Loop register	What rung is each loop on, and what evidence moves it?	Platform lead, loop owners	`loop-register.yaml`
Tooling portfolio (this page)	Which capability do we buy, build or trial to unblock a gate, and when do we stop?	CTO, one owner per item	`tooling-portfolio.yaml`
Business case	Is the whole program worth funding?	CTO, finance	Business case

What does each answer to Q24 lack?

Q24 sits in the Strategy and ROI section of the CTO Scorecard. Find your answer and fix the gap in its row. Each row is one step.

Your answer	Points	What is missing	Next step
No — reactive	0	Any list of intended capability	Write down the constraints blocking your pilot loops’ next gates; each becomes a candidate item
“When we have time” backlog	1	An owner, a budget and a review date	Convert each backlog line into the template below; delete lines that name no loop
Roadmap exists, has an owner, quarterly review	2	Hypotheses, measurable graduation and stop gates, an explicit budget	Add `hypothesis`, `decision_rule` and `budget` to every item, signed before the trial starts
Owned roadmap with capability hypotheses, measurable graduation and stop gates, quarterly review, and an explicit budget	3	Nothing for Q24; keep it honest	Show the stopped items at every review, and link each item to the loop register

The portfolio-item template

One file, owned by the platform team, reviewed like code. Each entry is one bet. The example tests whether browser evidence from the Playwright MCP server (@playwright/mcp, 0.0.82 on npm, checked 2026-09-26) lets a UI loop drop line-by-line diff review.

# tooling-portfolio.yaml — one entry per capability bet; reviewed quarterly
- id: browser-evidence-for-ui-fixes
  owner: jane.doe                      # one named person, never a team alias
  serves_loop: web-app/ui-bug-fixes    # must exist in loop-register.yaml
  unblocks: L3 -> L4                   # the gate this bet helps the loop pass
  constraint: >
    UI fixes wait for a human to click through the change; median time in
    review is the loop's largest delay.
  hypothesis: >
    If the agent attaches a browser trace and screenshots of every acceptance
    scenario to the evidence bundle, reviewers can approve on evidence and
    accepted lead time falls without more escaped UI defects.
  capability: browser verification (Playwright MCP in CI and local sessions)
  scope: { repos: [web-app, admin-ui], risk_class: low, window_weeks: 6 }
  outcome_metric: accepted_lead_time_median
  guardrails: [escaped_ui_defects, revert_rate, time_in_review, data_policy_events]
  budget:                              # full cost, see strategy/economics
    licenses_usd: 0
    usage_usd_cap: 1500
    ci_minutes_cap: 20000
    setup_and_review_hours: 60
    exit_hours: 8
  decision_rule: >
    Graduate if median accepted lead time is at least 20% below the loop's
    baseline over 30 merged changes and no guardrail is worse than baseline.
    Stop if any guardrail is worse for two consecutive weekly checks or the
    usage cap is reached before week 4. Otherwise narrow or continue once.
  signed_before_start: 2026-10-01
  exit_plan: remove MCP config and CI job, revoke test-account credentials, keep traces
  status: trial                        # candidate | trial | graduated | narrowed | stopped
  next_review: 2026-11-15

The thresholds (20%, 30 changes, two weekly checks) are starting values to adapt, not research findings. Set them from your loop’s own baseline and variance with pilot design, and sign them before the window opens: a threshold written after the results arrive proves nothing.

What each field must hold, and when to reject an item

Field	Holds	Reject the item when
`serves_loop`, `unblocks`	A loop id from the register and one ladder transition	The loop is not registered, or the item skips a rung
`constraint`	A delay, failure or cost observed in delivery data	The source is a vendor launch or a demo
`hypothesis`	A change in one outcome metric, caused by one capability	It names a tool but no measurable change
`guardrails`	Quality, security and data measures that must not get worse	Only speed or usage measures are listed
`budget`	Licenses, usage, CI, setup and review time, and exit work	Only the license line is filled in
`decision_rule`	Graduate and stop thresholds, dated before start	Thresholds are missing or dated after the trial began
`exit_plan`	What to revoke, remove and keep if the bet stops	It says “n/a”

The cost terms follow cost per accepted change, so a portfolio item and the program’s ROI evidence use one definition. The metric definitions come from metrics frameworks.

How to run one bet from candidate to decision

Start from the loop register. Take the loops whose next gate is blocked. Write the blocking constraint from delivery data: review queues, failed CI runs, incidents, onboarding friction and cost. A vendor announcement can suggest a capability, never a constraint.
Fix the evaluation before choosing the tool. Write hypothesis, outcome_metric, guardrails and decision_rule first. Only then compare candidate tools against that contract. Tools may change during the window; the decision contract does not.
Budget the whole experiment. Include security and legal review proportional to the data the tool touches (procurement questionnaire), integration, training, measurement, and exit work. Put a hard cap on usage spend per item.
Sign and register. The item owner, the loop owner and the CTO sign the entry through a reviewed pull request to tooling-portfolio.yaml. The merge date is the signed_before_start evidence.
Review on cadence and on trigger. Review every item at the quarterly gate review, and earlier when pricing, terms, data routing, a major capability or a risk changes. Record the decision and the evidence in the entry.
Act on the decision the same week. Graduate, narrow or stop with the table below. A decision recorded but not executed leaves a stopped item that still costs money.

Graduate, continue, narrow or stop: which decision applies?

Evidence at review	Decision	What changes
Outcome met the threshold, every guardrail at or better than baseline	Graduate	The platform team takes operating ownership; the tooling policy lists it as a default; controls, support and a retirement criterion are added; the loop register records the new capability
Outcome partly met, guardrails held, sample too small	Continue once	Extend the window once with the same thresholds; a second “continue” is a stop
Outcome met in one repository or risk class only	Narrow	Rewrite the scope to where it worked; start a new window with a new decision rule
A guardrail breached twice, the usage cap hit early, or no measurable change	Stop	Run the retirement checklist; record the reason; keep the evidence

A healthy portfolio shows stopped items. If every bet graduates, the thresholds are too loose or the review is not reading the evidence.

Retire a stopped bet so nothing stays live

Revoke API keys, service accounts, OAuth grants and test-account credentials the item created
Remove MCP server entries, CI jobs, hooks, plugins and managed-setting exceptions
Cancel seats or usage commitments; confirm the last invoice with finance
Export useful artifacts (prompts, skills, eval cases, traces) to the repository
Keep the decision evidence and link it from the entry
Update the tooling policy and the team guidance that mentioned the tool

Vendor-level exit, data export and fallback testing are covered in vendor risk management and avoiding lock-in.

Where each bet’s cost and usage evidence comes from

The template is the same for every tool. What differs is where the budget actuals come from. The full export instructions per tool live in the economics guide.

Take usage from your plan’s admin reporting and record the plan and billing mode in the item. Cursor’s plans and admin reporting were not verifiable from cursor.com on 2026-09-26, so confirm what your console reports before you write a usage cap into a decision rule.

Cap every headless run the bet adds to CI with claude -p --max-budget-usd 5 "…"; the flag works only with --print (checked against Claude Code 2.1.283). For per-user spend, export OpenTelemetry cost metrics (claude_code.cost.usage) into your own stack or route usage through a gateway; the Team and Enterprise analytics dashboard shows adoption and PR attribution, not per-user cost (cost governance).

Run CI steps with codex exec --json and store the JSONL events with the pull request number; add --output-schema when the bet needs a machine-readable result (checked against Codex CLI 0.157.1). Record whether CI authenticates with an API key (codex login --with-api-key) or a ChatGPT sign-in, because usage lands on a different account in each case; add both to the usage line.

Prompts for building and reviewing the portfolio

Read loop-register.yaml and tooling-portfolio.yaml. For the loop "web-app/ui-bug-fixes",
draft one new tooling-portfolio.yaml entry that unblocks its next gate.
Use this constraint from our delivery data: REVIEW_QUEUE_NUMBERS.
Fill every field of the existing entries' schema. Do not choose a vendor yet: write
the capability generically. The decision_rule must name graduate and stop thresholds
relative to the loop's baseline in loop-register.yaml, and the budget must include
usage, CI, setup and review hours, and exit work, not only licenses.
Mark any value you cannot derive from the files as TODO instead of estimating it.
Output the YAML only; do not edit any file.

Replace REVIEW_QUEUE_NUMBERS with the measured constraint for that loop, for example the median review wait and the CI failure rate over the last 30 days, with their source.

Read tooling-portfolio.yaml and loop-register.yaml. Flag every portfolio item where:
- serves_loop is not a loop id in loop-register.yaml;
- unblocks skips a rung or does not match the loop's current_rung;
- owner is a team alias rather than a named person;
- hypothesis names a tool but no measurable change in outcome_metric;
- guardrails lists only speed or usage measures;
- budget has no usage cap or no exit work;
- decision_rule is missing a graduate or a stop threshold;
- signed_before_start is later than the trial start, or next_review is past;
- two items test the same capability for the same loop.
Output one table: item id, violation, suggested fix. Then count items by status,
including stopped. Do not modify any file.

Prepare the review of portfolio item "browser-evidence-for-ui-fixes".
Read its entry in tooling-portfolio.yaml: decision_rule, guardrails, budget.
Use the GitHub CLI to list merged PRs for its loop since signed_before_start (2026-10-01):
  gh pr list --state merged --label loop:ui-bug-fixes --limit 100 \
    --search "merged:>=2026-10-01" \
    --json number,createdAt,mergedAt,reviews,files
Use only PRs merged on or after signed_before_start; drop any earlier PR.
Compute median lead time (created to merged) and compare it with the loop baseline
in loop-register.yaml. For each guardrail, report the value and the PR numbers behind it,
or "unknown" if it cannot be computed from these sources.
Recommend graduate, continue, narrow or stop strictly by the decision_rule. Separate
observed facts, assumptions and open risks. Do not edit any file.

The last prompt needs the GitHub CLI (gh) authenticated and a loop:* label on each loop’s pull requests. It runs in Claude Code, Codex or Cursor’s agent without changes.

How is the portfolio verified without reading every entry by hand?

Treat tooling-portfolio.yaml like code and let checks carry the load:

Schema and cross-reference check in CI. On every pull request that touches the file, validate tooling-portfolio.yaml against a JSON Schema and run a small script that cross-references each item against loop-register.yaml; make both a required check. An item that names an unregistered loop, lacks a stop threshold or has no exit plan fails the check and cannot merge. The audit prompt runs alongside as an advisory review, not as the gate, because a model’s verdict can differ between runs.
Signatures in the history. The merge of the entry, reviewed by the item owner, the loop owner and the CTO, dates the decision rule. A later edit to a threshold shows up as a diff and needs the same reviewers.
Reconciliation with finance. Once a quarter, finance matches each item’s usage and license actuals against invoices. An API key billed outside the tools’ own reports is the usual gap.
Decisions cross-checked with the loop register. A graduated item must appear as a capability of its loop; a stopped item must have its retirement checklist ticked.

Sign-off: the item owner is responsible for the evidence, finance signs the cost lines, and you, as CTO, are accountable for every graduate and stop decision.

What goes wrong with an AI tooling roadmap?

The roadmap is an inventory. A list of Claude Code, Cursor, Codex, MCP and “agents” with target dates cannot say what improves or when to stop. Recovery: rewrite each line with the template; any line that names no loop and no gate leaves the roadmap.

A second roadmap competes with the transformation roadmap. Tooling dates drive the plan, and loops wait for tools instead of the other way round. Recovery: sequence items only by the loop gates they unblock, and review them in the same quarterly gate review.

Sunk cost keeps a bet alive. The item has been “promising” for three quarters. Recovery: apply the one-continue rule; a second continue is a stop, and a new idea gets a new entry with a new decision rule.

Graduation without an owner. The trial ends well, the enthusiast moves on, and nobody patches the MCP server or renews the account. Recovery: graduation is not complete until the platform team has accepted operating ownership in writing, as the agent platform team charter describes.

A stop that is only a status change. The entry says stopped, but the service account still has repository access. Recovery: run the retirement checklist and link its evidence before the status can change.

Two bets test the same capability. Two teams trial two browser-verification tools for the same loop and neither reaches a sample size. Recovery: merge them into one item with one decision rule, or split them by loop.

Where to go next after Q24

Q23 comes before this question and supplies the ROI evidence; Q25 checks that every graduated bet has a tested exit.

The organization-wide transformation roadmap The stages, loop register and ladder gates each portfolio item serves.

Adoption roadmap for one repository The per-repository slice where the loop gates are run.

Q23 · AI tooling ROI The evidence pack a graduate decision depends on.

Q25 · Vendor risk management Test fallback, degraded mode, export and recovery for graduated tools.

Frequently asked questions

What belongs in an AI tooling roadmap portfolio item for CTO Scorecard Q24?

Each roadmap item defines one funded capability bet targeting a specific delivery constraint in a named loop, complete with a falsifiable hypothesis, a full-cost budget, quality guardrails, and a dated stop-or-graduate review gate.

Should an organization maintain a separate AI roadmap alongside its engineering transformation roadmap?

No, AI tooling bets are the tooling layer of the organization single engineering transformation roadmap rather than a parallel roadmap disconnected from delivery constraints.