Prototype policy — graduate by risk and evidence

Level 2 How-to Updated 2026-10-12

A prototype policy for AI-generated software classifies every prototype by exposure (users, data, external effects, and reversibility), not by the tool that built it. It then requires six kinds of evidence before any prototype graduates: data approval, a named owner, tests, security review, observability, and a rehearsed rollback. That combination earns the full three points on CTO Scorecard question 16.

This page is for the CTO who owns question 16 and for the platform or tech lead who has to make the policy real. The usual situation: an engineer vibe-coded a pricing calculator in an afternoon to settle an argument with sales. Four months later, account managers send its link to customers, it reads a copy of the CRM, and it runs on the engineer’s personal cloud account. Nobody decided it was production. It became production one user at a time.

Q16 · Parallelism at team scale: How does the team classify and graduate rapidly generated prototypes?

Max-score answer: a risk-tier policy with data, ownership, test, security, observability, and rollback criteria before any prototype graduates.

What this prototype policy gives you

A scoring table for Q16 that shows what each answer lacks and the one change that moves you up.
Four exposure classes with the technical boundary that enforces each one, not only a label.
A prototype.yml register file and a CI check you can commit as they are.
A graduation trigger table that says how each trigger is detected without anyone remembering to look.
A six-criterion graduation gate that states the evidence and the person who signs each line.
Three copy-paste prompts: classify a prototype, sweep for fired triggers, and audit a graduation request.

Prototypes built by analysts, operations staff, and other people outside engineering need extra controls (app-builder procurement, intake forms, and red lines for non-engineers). Those live on software built outside engineering, which uses the same four classes. This page covers the policy itself and how engineering enforces and audits it.

How does each Q16 answer score?

The scorecard awards 0–3 points. Each step up replaces an intention with a control.

Points	Answer	What it lacks	Move to the next level
0	Nobody uses them / no policy: grey zone	Nobody knows how many prototypes exist or what data they touch	Start the register below and run one discovery pass
1	Banned, no exceptions	Engineers still prototype, on personal accounts you cannot see	Replace the ban with the disposable class: sanctioned, sandboxed, and expiring
2	Allowed for prototypes / marketing landing pages, not for production	“Not production” has no definition, so nobody notices the day it stops being true	Define classes by exposure and automate the graduation triggers
3	Risk-tier policy with data, ownership, test, security, observability, and rollback criteria before any prototype graduates	Nothing on the answer sheet; the remaining risk is a policy that exists only on paper	Prove it quarterly with the verification routine below

A ban scores above no policy because it at least states intent. It scores below “allowed for prototypes” because it pushes the work where you cannot inspect it. Andrej Karpathy puts the line this policy draws: “Vibe coding raises the floor. Agentic engineering is about extrapolating the ceiling,” and “You are still responsible for your software, just as before” (his published transcript of his Sequoia Ascent talk, 2026-04-30). Simon Willison proposed vibe engineering as a name for the accountable side of that line; he later wrote that agentic engineering was winning.

Which class does a prototype belong in?

Classify by what the prototype can reach and affect, never by its age, its size, or the tool that wrote it. A 200-line script that writes to the billing database is not a prototype in any sense that matters.

Class	Boundary	Technical enforcement	Evidence required
Disposable exploration	Synthetic or approved sample data, no users beyond the author’s team, no writes to shared systems	Separate sandbox account or project, no production credentials issued, egress limited to package registries, no public URL	Purpose, owner, and expiry in the register
Controlled pilot	Named users inside one team, approved data, reversible effects	Company repository with CI, a scoped service account, SSO in front of the app, a spending cap	Accepted spec, tests that run in CI, basic monitoring, a written rollback, a named engineering owner
Production system	Other teams or customers depend on it, persistent data, or material external effects	The standard production path: deploy pipeline, secrets manager, on-call rotation	The full AI-native SDLC loop and the graduation gate below
Forbidden path	Secrets in code, unapproved personal or regulated data, payments, destructive access to shared systems	Blocked by data-access controls and managed agent settings, not by the prototype author’s judgment	None: stop and move the work to the production path

The four classes line up with the four risk tiers in governance and autonomy. Disposable work runs at Tier 0–1, a controlled pilot touches Tier 2, and anything in the production class is Tier 3 as soon as it handles authentication, billing, or production data, however small the code is.

How do you roll out the prototype policy?

Publish the classes and the forbidden list. Put the table above into your AI usage policy as its own clause. State that the class follows exposure and that nobody needs approval to start a disposable prototype. Removing that friction is what keeps prototypes visible.
Give the disposable class a sanctioned home. Create a sandbox cloud account or project, a repository template with the register file and CI check below, and a sample dataset. If the sanctioned path is slower than a personal account, people will use the personal account.
Register every prototype on creation. The prototype.yml file at the repository root is the register entry. A central inventory is a scheduled job that collects these files across the organisation, not a spreadsheet someone maintains.
Automate the graduation triggers. Wire each detection signal in the trigger table to an alert or a failing check. A trigger that depends on the author noticing will not fire.
Run graduation as a gate, not a conversation. A prototype enters the production class only when every line of the graduation gate has its evidence and its signature. Record the decision to harden in place or rebuild, with the reason.
Retire on expiry and verify the cleanup. An expired prototype is renewed with a new date or retired. Retirement revokes credentials, deletes data copies, removes domains and DNS records, closes billing, and archives the repository with its decision record.

What goes in the prototype register?

Commit this file at the root of every prototype repository. The CI check below fails when a field is missing or the expiry has passed, which turns “we should review old prototypes” into a red build.

name: pricing-calculator
class: controlled-pilot        # disposable | controlled-pilot | production | forbidden
owner: anna.nowak              # a person, never a team alias
engineering_owner: team-revenue-platform   # required from controlled-pilot upward
purpose: Let account managers quote volume discounts without a spreadsheet.
users: account-managers-emea (14 people)
data_read: [crm-replica:internal]
data_written: none
external_effects: none         # email, payments, third-party API writes, or none
credentials: svc-pricing-calc (read-only, secrets manager)
public_url: false
expiry: 2026-12-15
last_trigger_review: 2026-09-26

Add this workflow to the prototype repository template. It needs no secrets and only reads the repository.

name: prototype-register
on:
  pull_request:
  schedule:
    - cron: '17 6 * * 1'   # every Monday, so an idle prototype still expires
permissions:
  contents: read
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - name: Check class, owner, and expiry
        run: |
          test -f prototype.yml || { echo "prototype.yml is missing"; exit 1; }
          for key in class owner expiry; do
            grep -Eq "^${key}: *[^ #]" prototype.yml || { echo "prototype.yml has no ${key}"; exit 1; }
          done
          if grep -Eq '^class: *forbidden' prototype.yml; then
            echo "Forbidden class: move this work to the production path"; exit 1
          fi
          if grep -Eq '^class: *(controlled-pilot|production)' prototype.yml; then
            grep -Eq '^engineering_owner: *[^ #]' prototype.yml || { echo "engineering_owner is required for this class"; exit 1; }
          fi
          expiry=$(sed -n 's/^expiry: *\([0-9-]*\).*/\1/p' prototype.yml)
          [[ "$expiry" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]] || { echo "expiry must be an unquoted YYYY-MM-DD date"; exit 1; }
          if [[ "$(date -u +%F)" > "$expiry" ]]; then
            echo "Expired on ${expiry}: renew with a new date or retire"; exit 1
          fi

The scheduled run matters more than the pull-request run. A prototype nobody touches is the one that expires silently. GitHub sends a failing scheduled run’s notification only to the user who last modified the cron line, so route the failure to the register owner as well, for example with a step that opens an issue assigned to owner. In a public repository, GitHub disables scheduled workflows after 60 days without repository activity, so keep prototype repositories private or have the central inventory job flag disabled workflows.

Which events should trigger a graduation review?

A graduation trigger is any change in exposure. Each one needs a detection signal that does not rely on the author.

Trigger	Detection signal	Response
Users outside the owning team	SSO or access logs show new groups; the app appears in a shared channel or wiki	Review within one week; the class moves to controlled pilot at least
Customer, personal, or regulated data	Data-access request, a new connection to a classified store, or a schema field matching your PII patterns	Stop until privacy and data handling approves the data use
Writes to a shared system or durable state	A new database migration directory, write scopes requested on a service account	Graduation gate before the next release
External effects: email, payments, third-party writes	A payment or email SDK in the dependency list; outbound calls to new domains	Forbidden until it passes the gate; payments go straight to the production path
Another system depends on it	Another repository calls its API or imports its package; it shows up in a service catalogue	Graduation gate; an engineering owner is named that week
Public exposure	A new DNS record, a public URL, or removed SSO	Take it private first, then review
Support obligation	Someone other than the author is paged or asked to fix it	Graduation gate; the prototype has become production already

Age is not a trigger. A two-year-old internal notebook with synthetic data can stay disposable forever if it keeps renewing its expiry. A two-day-old app with a customer login cannot.

What must a prototype prove before it graduates?

The graduation gate checks the six criteria in the Q16 max-score answer. Each criterion has an evidence item that a reviewer can check without reading the code line by line, and a named person who signs it.

Criterion	Evidence that proves it	Who signs
Data	Every store it reads and writes is classified, the classification permits this use, and no copy of production data lives outside approved storage	Data owner or privacy lead
Ownership	An engineering team owns the service, it is on that team’s on-call rotation, and the original author has handed over the behaviour notes	Engineering manager of the owning team
Tests	Acceptance criteria derived from real examples of current behaviour, each covered by a test that runs in CI and fails when the behaviour breaks	Tech lead of the owning team
Security	Authentication and authorisation on every route, no secrets in code or client bundles, dependencies confirmed to exist on the public registry, a security review of the diff with no open high findings	Security reviewer
Observability	Error tracking, logs with a retention setting, and one alert on the failure users would notice first	Owning team’s on-call lead
Rollback	A written rollback or kill switch, rehearsed once in a non-production environment, with the time it took	Owning team’s on-call lead

The owning team then submits the change like any other, with the evidence bundle attached. The graduation record states harden in place or rebuild, and why. Harden when the code can move to a company repository, the stack matches what your platform runs, and the security review finds isolated issues. Rebuild when the security findings are systemic or the data model will not hold real volume. In a rebuild, keep the acceptance criteria and drop the code: the prototype has already paid for discovering what the software should do. Software built outside engineering has the full harden-or-rebuild table.

How do you run the prototype review in each tool?

The prompts in the next section work in any of the three agents. What differs is how you keep a review session read-only and where the second opinion comes from.

Start the review read-only with claude --permission-mode plan, paste the prompt, and let the agent report without editing. For the security criterion, run /security-review on the branch diff, and /code-review once the hardened version is a pull request.

From v2.1.283 (both release channels on 2026-10-12), interactive Claude Code sessions in the terminal and VS Code start in auto mode, where a classifier approves actions instead of the engineer, unless a settings file sets disableAutoMode or the session’s model does not support auto mode; claude -p starts in Manual (auto mode on third-party providers or with telemetry off, from v2.1.285). That is reasonable in the disposable sandbox. To stop it organisation-wide, set permissions.disableAutoMode: "disable" in managed settings (checked on v2.1.283).

Run the review with a read-only permission profile: codex -c default_permissions=":read-only" (permission profiles are beta, checked on Codex CLI 0.157.1). For the pull request of the hardened version, codex review --base main reviews the branch non-interactively and fits as a CI step.

Admins pin the permission profile and restrict MCP servers for everyone through requirements.toml, so a prototype session cannot reach an unapproved server. See managed policy for the file.

Copy-paste prompts for prototype classification and graduation

Run the first prompt when a prototype is created or found, the second on a schedule, and the third when someone requests graduation. All three are read-only.

Read this repository without modifying any file. Determine:
- who can reach it (auth, SSO, public URLs, deploy config);
- every data store it reads or writes, and whether any field looks like
  personal, customer, or payment data;
- every external effect (email, payments, third-party API writes);
- every credential it uses and where that credential is stored.

Then classify it as disposable, controlled-pilot, production, or forbidden,
using these rules: forbidden if secrets are in code, payments are possible,
or personal data is used without an approval reference; production if other
teams or customers depend on it or it writes durable shared state;
controlled-pilot if named users inside one team use it on approved data;
otherwise disposable. Output a filled prototype.yml and, separately, the
file and line that justify each field.

Compare the current state of this repository with prototype.yml. Report,
with file and line or commit as evidence, every graduation trigger that has
fired since last_trigger_review:
1. users or groups beyond those listed;
2. new data stores, migrations, or write scopes;
3. new dependencies that send email, take payments, or call third-party APIs;
4. new public URLs, DNS records, or removed authentication;
5. other repositories or services that call this one.
If none has fired, say so and list what you checked. Do not edit files.

Audit this graduation request against six criteria: data, ownership,
tests, security, observability, rollback. For each criterion, state PASS,
FAIL, or MISSING EVIDENCE, and cite the file, CI run, or document that
supports it. Specifically check that every acceptance criterion maps to
a test that runs in CI, that no secret appears in code or client bundles,
that every dependency exists on its public registry, and that the rollback
note names a command or switch and a rehearsal date. End with the
missing items only, ordered by risk. Do not change any code.

How do you prove the policy works without reading every prototype?

The evidence is structural, so the CTO checks counts and exceptions rather than code.

Register coverage. Compare repositories and cloud projects found by discovery (source control, SSO application lists, cloud billing) with repositories that contain prototype.yml. Every gap is either registered or retired.
Red builds on expiry. Count scheduled register checks that failed and were resolved within two weeks. A failing check nobody resolves means the owner field is wrong.
Graduation records. Every service in the production class that began as a prototype has a gate record with six signatures and a harden-or-rebuild decision.
Unowned exposure. The number of prototypes with users outside the owning team and no engineering_owner should be zero at every quarterly review.
Retirement completeness. For a sample of retired prototypes, confirm that the credentials no longer authenticate and the domains no longer resolve.

Report these five quarterly. They are also the evidence a scorecard reviewer asks for when you claim three points on Q16.

What breaks in a prototype policy, and how do you recover?

“Temporary” production. A prototype gains users, data, and a dependency, and nobody declares it production. Recovery: treat the support-obligation trigger as final. If someone other than the author fixed it this month, run the graduation gate now and name an owner this week.

The register becomes a form nobody fills in. Engineers skip it because it is separate from the work. Recovery: make the file part of the repository template and the CI check part of the default branch protection, so creating a prototype creates its register entry.

The disposable sandbox is too restrictive. Engineers move back to personal accounts. Recovery: measure how long it takes to start a sanctioned prototype, and remove approvals from the disposable class entirely. Keep the controls on credentials and data, not on starting work.

Graduation takes a quarter. Teams stop requesting it and let prototypes drift into use. Recovery: publish the median time from trigger to gate decision, and staff a small rotation of engineering owners who can take on a graduated service.

Hardening in place carries the prototype’s shortcuts into production. Recovery: the security and data criteria are pass or fail, not advisory. A systemic finding (missing authorisation, secrets in history, personal data in logs) switches the decision to rebuild.

Retirement leaves live credentials. The repository is archived, but the service account and the DNS record remain. Recovery: retirement is complete only when a check proves the credential fails and the domain does not resolve, recorded in the register.

Where to go next with your prototype policy

Software built outside engineering The same four classes for analysts and operations staff, with app-builder procurement and red lines.

Governance and autonomy The four risk tiers and the human decision each one requires.

The evidence bundle What a graduating change must prove in its pull request.

AI usage policy template Where the prototype clause lives, and how its other clauses are enforced.

Back to the CTO Scorecard answer key for the other questions in the Parallelism at team scale section.

Frequently asked questions

How should a prototype policy classify rapidly generated AI prototypes?

Classify every prototype by its exposure—who can reach it, what data it touches, what external effects it can trigger, and how reversible it is—across four exposure classes rather than by which tool generated the code.

What six criteria must a prototype pass before graduating to production?

Before graduation, a prototype requires signed data-class approval, a named long-term code owner, automated test coverage, a security review, production observability, and a rehearsed rollback.