Rehearsed rollback with human production authority

Level 2 How-to Updated 2026-10-12

A rehearsed rollback pipeline reverses a harmful production change with one protected command. A versioned contract names the target release, preconditions and recovery checks, and a deploy job with short-lived credentials executes it. Coding agents draft the decision and verify recovery, while a named human, or a deterministic alert rule for pre-approved low-risk services, starts the rollback.

It is 16:40 on a Friday. The release an agent wrote at lunch passed every check, and checkout errors have just tripled. Someone opens a terminal to run git revert, someone else asks whether last week’s migration is reversible, and nobody remembers when the rollback script last ran. This page is for the developer who owns the release pipeline and the tech lead who carries the pager. It turns rollback from an improvisation into the most rehearsed path you have.

Scorecard question (Q21): How are rollbacks managed if a deployment introduces a regression?

Maximum-score answer: Rehearsed single-command rollback exposed via scoped MCP tools, capable of being triggered deterministically when control bands breach.

“Scoped” is the operative word. On this page the agent’s MCP tools read deployment state and draft the decision; the command itself runs in a protected job, and it is started by a person or by a deterministic alert rule, never by a model’s judgment. That keeps the production approval gate intact during an incident, which is when people most want to bypass it.

What a rehearsed rollback pipeline gives you

A rollback contract in YAML that sits next to the deploy workflow and names the target, preconditions, abort conditions and recovery checks.
A GitHub Actions rollback job that runs through the same approval gate as a deploy, with a release ID as its only required input and no standing credentials.
A read-only agent setup for Claude Code, Codex and Cursor that drafts the rollback decision record while the incident channel is still assembling.
Three copy-paste prompts: the decision record, the recovery verdict and the rehearsal audit.
A rehearsal routine and a readiness checklist that prove the path works without anyone reading the script line by line.

Why rollback matters more once agents write the code

Agents raise the volume of change, and every change is a chance to break production. The 2025 DORA report found both sides of that trade: “we observe a positive relationship between AI adoption on both software delivery throughput and product performance”, and “AI adoption does continue to have a negative relationship with software delivery stability” (Google Cloud / DORA, 2025-09-23). DORA names the mechanism as missing control systems such as “strong automated testing, mature version control practices, and fast feedback loops”.

A rehearsed rollback is one of those controls. It is also what lets you stop reading every diff: when a bad change costs minutes of partial impact instead of an evening of hotfixes, you can trust tests, progressive delivery and recovery checks to carry the risk.

Should you roll back, roll forward or turn off a flag?

A rollback is not automatically the safe option. Decide by what the change touched, and write the answer into the contract before the incident, not during it.

The harmful change…	Prefer	Why
Sits behind a feature flag	Turn the flag off	Takes effect in seconds, needs no deploy, and leaves the release in place
Is stateless code or config, and the previous release is retained	Roll back to the previous immutable release	Fastest full reversal; this page’s main path
Includes an expand-only migration (new nullable column, new table)	Roll back the code; leave the schema	The old release ignores the new schema
Includes a contract migration (drop, rename, type change) or rewrote data	Roll forward with a fix, or run a tested data-restore plan	The old release cannot read the new data
Changed queue message formats, cache formats or a public API that clients already use	Roll forward, or roll back only if both formats are readable	Old consumers choke on messages the new release already produced
Shipped to mobile clients	Flag off server-side and roll forward	Installed apps cannot be rolled back

Canary aborts and flag kill switches belong to progressive delivery. This page covers the release-level rollback that stays in reserve when those are not enough.

How do you build a one-command rollback?

Write the rollback contract next to the deploy workflow. Commit it as deploy/rollback.yml, and review it like code. The example below is illustrative; replace the command and checks with the tested interface of your platform.

# deploy/rollback.yml — reviewed by the service owner
service: checkout-api
risk_class: standard               # low | standard | high, as in progressive delivery
target: previous-retained-release  # an immutable, signed release ID, never a branch
command: ./scripts/rollback.sh --release "$RELEASE_ID"
dry_run: ./scripts/rollback.sh --release "$RELEASE_ID" --dry-run
executor_identity: github-environment:production   # the same gate as a deploy
approver_role: incident-commander
auto_trigger:                       # deterministic, never model-decided
  allowed: false                    # true only for low-risk, stateless services
  alert_rule: checkout-error-budget-burn-fast
preconditions:
  - current release equals EXPECTED_RELEASE
  - target release artifact is retained and its signature verifies
  - no migration since target is marked "contract"
abort_conditions:
  - target release has an active security block
  - a data migration since target is not backward compatible
recovery_checks:
  - error-budget fast-burn alert clears and stays clear for 10 minutes
  - checkout synthetic passes from two regions
  - p95 latency within the pre-incident band
last_rehearsal: 2026-09-18          # updated by the rehearsal job, not by hand
evidence: incident record + deployment audit log

Three fields do most of the work. preconditions and abort_conditions make the risky cases from the decision table machine-checkable. recovery_checks defines “recovered” before anyone is under pressure. last_rehearsal lets CI refuse a release when the rollback path has gone stale.

Make the script refuse, not guess. scripts/rollback.sh checks every precondition and abort condition before it changes anything, and exits non-zero with the reason when one fails. --dry-run runs the same checks and prints the plan. The script never picks its own target: it takes one release ID and verifies it.

Run the script only inside a protected job. Add a workflow_dispatch workflow that runs in the same production environment as your deploy, so the required reviewer, the OIDC trust policy and the audit trail from Production approval as a system boundary apply unchanged.

name: rollback
on:
  workflow_dispatch:
    inputs:
      release_id:
        description: Immutable release ID to restore
        required: true
        type: string
      target_env:
        description: Environment to roll back
        required: true
        type: choice
        options: [staging, production]
        default: production
permissions:
  contents: read
concurrency:
  group: ${{ inputs.target_env }}-deploy   # the deploy workflow's group: never race a deploy
  cancel-in-progress: false
jobs:
  dry-run:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write                # read-only cloud credentials for the precondition checks
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      # Add a read-only OIDC login here (a role that can list releases and read the registry,
      # nothing more), so the preconditions can check the current release and the target artifact.
      - name: Validate input and preconditions
        env:
          RELEASE_ID: ${{ inputs.release_id }}   # passed as env, never interpolated into the script
        run: |
          [[ "$RELEASE_ID" =~ ^[a-z0-9._-]{1,64}$ ]] || { echo "invalid release id"; exit 1; }
          ./scripts/rollback.sh --release "$RELEASE_ID" --dry-run | tee -a "$GITHUB_STEP_SUMMARY"

  rollback:
    needs: dry-run
    runs-on: ubuntu-latest
    environment: ${{ inputs.target_env }}   # waits for that environment's required reviewer
    permissions:
      contents: read
      id-token: write                # short-lived cloud credentials after approval
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      # Add your cloud's OIDC login step here; each role trusts only its own environment subject.
      - name: Roll back
        env:
          RELEASE_ID: ${{ inputs.release_id }}
        run: ./scripts/rollback.sh --release "$RELEASE_ID"

The target_env input exists for rehearsals: the same file runs against staging, whose environment has its own required reviewer and its own OIDC role. The dry run gets only read-only credentials and lands in the job summary, so the approver sees the plan on the same page as the Review deployments button. No agent step runs in this workflow.

The shared concurrency group keeps a rollback from running alongside a deploy, but it does not queue them: GitHub Actions keeps at most one pending run per group. A later run in the same group cancels a pending one, so pause automated deploys first (gh workflow disable deploy.yml) and confirm the rollback run is queued, not cancelled.

Give production (and staging) the same deployment-branch rule, default branch only, so the approved job always runs the reviewed scripts/rollback.sh. Without it, anyone with write access can dispatch rollback.yml from their own branch, and the reviewer approves a script nobody reviewed.

Wire the deterministic trigger, only where the contract allows it. The production environment always waits for its required reviewer, so an alert that dispatches rollback.yml only queues a request: the alert pages the on-call engineer, who approves. That is the default, and it is the only path for a service whose contract says auto_trigger.allowed: false. For a service whose contract says true, give the automatic path its own door instead of weakening the shared one:
- A second workflow, .github/workflows/rollback-auto.yml, with the same steps as rollback.yml, bound to a second environment, production-auto.
- production-auto has no required reviewer. Its protection is a deployment-branch rule that allows only the default branch, and a cloud OIDC trust condition that accepts only the environment:production-auto subject, and only for the roles of the pre-approved services.
- The alerting system (not an agent) dispatches it through a GitHub App whose token no person or agent holds, passing the release ID of the previous successful deployment. The job refuses any other caller and any service whose contract does not allow it.
```
# .github/workflows/rollback-auto.yml: only the differences from rollback.yml
concurrency:
  group: production-deploy                  # still never races a deploy
jobs:
  rollback:
    if: github.actor == 'alert-bridge[bot]'  # the alerting system's GitHub App
    environment: production-auto            # no reviewer: branch rule + OIDC trust only
    steps:
      # checkout, read-only login and dry run as in rollback.yml, then:
      - name: Refuse unless the contract allows automatic rollback
        run: test "$(yq '.auto_trigger.allowed' deploy/rollback.yml)" = "true"
```
The branch rule matters: without it, anyone who can push a branch could edit the if: line and reach the unreviewed environment. Deterministic anomaly detection covers how to make the alert rule replay-tested and versioned.
Make CI enforce rehearsal freshness. Add a step to the deploy workflow, before the deploy job, that reads last_rehearsal from the contract and fails when it is older than your policy, for example 30 days. yq is preinstalled on GitHub’s Ubuntu runners.

The gate also blocks the roll-forward hotfix that the decision table recommends during an incident, so give it one documented exit: a deploy dispatched with an incident ID skips the check, records the ID in the job summary and opens a rehearsal ticket automatically.
.github/workflows/deploy.yml
```
on:
  push:
    branches: [main]
  workflow_dispatch:
    inputs:
      incident_id:
        description: Incident ID for an emergency roll-forward (skips the freshness gate)
        required: false
        type: string
# ...in the job that gates the deploy (it needs `issues: write` for the ticket):
- name: Block the release when the rollback rehearsal is stale
  if: ${{ !inputs.incident_id }}
  run: |
    last=$(yq '.last_rehearsal' deploy/rollback.yml)
    if [[ $(date -d "$last" +%s) -lt $(date -d '-30 days' +%s) ]]; then
      echo "Rollback last rehearsed on $last; rehearse before shipping."; exit 1
    fi
- name: Record the incident bypass and open a rehearsal ticket
  if: ${{ inputs.incident_id }}
  env:
    INCIDENT_ID: ${{ inputs.incident_id }}
    GH_TOKEN: ${{ github.token }}
  run: |
    echo "Freshness gate skipped for incident $INCIDENT_ID" >> "$GITHUB_STEP_SUMMARY"
    gh issue create --title "Rehearse rollback after incident $INCIDENT_ID" \
      --body "Deploy run $GITHUB_RUN_ID skipped the rehearsal-freshness gate."
```
A stale rollback path then blocks routine shipping instead of surprising you mid-incident, and every bypass leaves a named incident and an open ticket behind. The gate reads a date from the repository, and on push the workflow itself also comes from the repository, so a pull request (an agent’s included) can move last_rehearsal or auto_trigger.allowed, or delete the step. Put deploy/rollback.yml, scripts/rollback.sh and /.github/workflows/ under CODEOWNERS for the service owner with a required code-owner review, and reject any last_rehearsal change that does not come from the rehearsal job’s pull request.

For a human-initiated rollback, the on-call engineer runs one command and approves in the platform:

# terminal, as the on-call engineer
gh workflow run rollback.yml -f release_id=2026-09-26.3 -f target_env=production

How do agents help without touching production?

The agent’s job is the part humans do badly at 16:40 on a Friday: gathering evidence, checking preconditions against the contract and writing a clear decision record. Give it read-only access to deployment state, for example the GitHub MCP server’s Actions toolset in read-only mode (https://api.githubcopilot.com/mcp/x/actions/readonly, per GitHub’s github-mcp-server README), and a token for a diagnostic identity that cannot dispatch workflows. The setup differs by tool.

Connect the read-only toolset with a token from your secret store, then run the decision prompt headless with an allowlist. The shell expands the token and Claude Code stores it in your local config; use a short-lived, read-only token and rotate it. dontAsk denies every tool call that would prompt, so the agent cannot run gh workflow run or gh api unless you allow them; mcp__github-actions-ro allows every tool on the read-only server.

# terminal, as the diagnostic identity
claude mcp add --transport http github-actions-ro \
  https://api.githubcopilot.com/mcp/x/actions/readonly \
  -H "Authorization: Bearer $GITHUB_READONLY_TOKEN"

claude -p --permission-mode dontAsk \
  --allowedTools "Read,Grep,Glob,mcp__github-actions-ro,Bash(gh run view *),Bash(gh run list *),Bash(gh release view *)" \
  --output-format json \
  "$(cat prompts/rollback-decision.txt)" > rollback-decision.json

Flags checked against claude --help, v2.1.283.

Register the same read-only endpoint with the token read from an environment variable, and run the prompt under the built-in read-only permission profile (beta).

# terminal, as the diagnostic identity
codex mcp add github-actions-ro \
  --url https://api.githubcopilot.com/mcp/x/actions/readonly \
  --bearer-token-env-var GITHUB_READONLY_TOKEN

codex exec -c default_permissions=":read-only" --ephemeral \
  -o rollback-decision.md "$(cat prompts/rollback-decision.txt)"

Do not add --sandbox to the same command: OpenAI says the profile and legacy sandbox systems do not compose. Flags checked against codex exec --help and codex mcp add --help, v0.157.1.

Add the read-only endpoint to your global ~/.cursor/mcp.json, which stays out of the repository. GitHub’s Cursor installation guide uses this url + headers shape with a literal token; do not paste one into the file. Cursor’s MCP docs describe ${env:NAME} interpolation in mcp.json, so export the diagnostic identity’s read-only token as GITHUB_READONLY_TOKEN from your secret store before you start Cursor. Confirm the interpolation in your build: if it is not expanded, the server fails to authenticate, which is safe.

{
  "mcpServers": {
    "github-actions-ro": {
      "url": "https://api.githubcopilot.com/mcp/x/actions/readonly",
      "headers": { "Authorization": "Bearer ${env:GITHUB_READONLY_TOKEN}" }
    }
  }
}

Restart Cursor, open the agent in Plan Mode, which plans before it writes any code, and paste the decision prompt. Cursor’s Run modes and Hooks control which shell commands the agent may run; the run modes are Auto-review, Allowlist and Run Everything, under Settings > Agents > Approvals & Execution (Cursor docs, checked 2026-10-12). Confirm the hook names and CLI flags in your build before you rely on them during an incident.

Keep the prompt in the repository (prompts/rollback-decision.txt) so every on-call engineer runs the same one, and so a change to it gets reviewed.

Prepare a rollback decision record for the current checkout-api incident.
Read deploy/rollback.yml, the last five production runs of the deploy and rollback
workflows, and the releases since the rollback target. Use read-only tools only.

Report, in this order:
1. Current release and proposed target release, with run IDs.
2. Each precondition and abort condition from the contract: met, not met, or unknown,
   with the evidence for each.
3. Migrations, message or cache format changes, and flags since the target, each marked
   reversible or not.
4. Recommendation: flag off, roll back, or roll forward, with the reason in two sentences.
5. The exact command a human would run, and the recovery checks to watch.
Do not dispatch workflows, approve deployments, or read any production credential.

The rollback run RUN_ID has completed. Compare production against each
recovery_check in deploy/rollback.yml, using the monitoring queries it names.
For every check, give the query, the time window, the value and a verdict:
recovered, not recovered, or inconclusive. Treat missing data as inconclusive, never
as recovered. List follow-ups (the failing test or eval to add, the migration to fix).
Do not declare the incident resolved; the incident commander does.

Replace RUN_ID with the ID that gh run list --workflow rollback.yml prints.

How do you rehearse a rollback honestly?

A rehearsal proves the path only if it uses the real path. Run it on a schedule, in staging or a production-shaped environment, with the same workflow, the same identities and an environment gate configured like production’s.

Deploy a known release to staging, then deploy a deliberately bad one that trips a recovery check, for example a synthetic that returns errors.
Run the decision prompt and confirm that the agent names the right target and flags every precondition correctly.
Dispatch rollback.yml with target_env=staging (gh workflow run rollback.yml -f release_id=<known-good> -f target_env=staging) and have a second person approve it. The staging environment has a required reviewer too, exactly as in production.
Run the recovery verdict prompt and compare its verdict with the dashboards.
Record the result: who ran it, the approver, start and end times, time to recover, the verdict, and what broke. The rehearsal job then opens a pull request that updates last_rehearsal in the contract.

The rehearsal job is a scheduled workflow (on: schedule, for example weekly) that runs steps 1, 3 and 5 itself: it deploys both releases, dispatches the rollback, pauses at the staging reviewer and opens the pull request with the recorded result. The on-call pair runs the two prompts in steps 2 and 4.

Audit our last rollback rehearsal against production reality. Read deploy/rollback.yml,
.github/workflows/rollback.yml, the rehearsal run log and the production deploy workflow.
Report every difference between rehearsal and production: identities, environment
protection, OIDC trust conditions, script flags, artifact retention, recovery-check
queries and data volumes. For each difference, say whether it could make a production
rollback fail when the rehearsal passed, and propose the smallest fix.
Change nothing; output a Markdown report.

How do you know the rollback worked without reading the script?

You verify behaviour, not code. Four pieces of evidence replace a line-by-line read:

The dry run lists every precondition and its result before the approver clicks.
The recovery checks in the contract return values, not opinions, and the verdict prompt reports them with timestamps.
The rehearsal record shows the same job, identity and gate succeeded recently, with a measured time to recover.
The audit log names who dispatched, who approved and which release ran.

The incident commander, not a model, declares recovery. After recovery, the gap that let the change through becomes a test or an eval, as When an agent causes an incident describes, and the incident re-enters the lifecycle through Maintain.

What breaks in a rollback pipeline, and how do you recover?

The target artifact is gone. Retention policy deleted the previous image, and the script has nothing to restore. Recovery: roll forward with a fix now; afterwards, retain at least the last few production releases, and make the rehearsal fetch the target from the production registry.

The migration is not backward compatible. The old release starts and fails on the new schema. Recovery: abort the rollback, and roll forward. Prevent it with expand-and-contract migrations, and mark every contract migration in the release so the precondition catches it.

The rehearsal used different permissions. Staging ran with an admin token, and production OIDC refuses the job. Recovery: dispatch the job with the break-glass path the incident process defines, then fix the rehearsal to use the exact environment and trust policy, and re-run the rehearsal audit prompt.

Deploy and rollback raced. An auto-merge deployed a new release while the rollback ran, and production ended on neither. Recovery: pause automated deploys (gh workflow disable deploy.yml), confirm no deploy run is pending, re-run the rollback, and keep the shared concurrency group on both workflows, as step 3 explains.

The recovery checks pass, but users still fail. The checks watched service health, not the user journey. Recovery: keep the incident open, add a synthetic for the broken journey to recovery_checks, and treat inconclusive signals as not recovered.

The agent was given write access to speed things up. A false alarm rolled back a healthy release at peak traffic. Recovery: revoke the token, reconnect the read-only endpoint, and move automatic triggering to a versioned alert rule that the contract permits.

Is your rollback path ready?

deploy/rollback.yml exists for the service, names an immutable target and was reviewed by its owner.
The rollback job uses the production environment gate and short-lived credentials, and no agent step runs in it.
The production and staging environments allow deployments from the default branch only, so an approval always runs the reviewed script.
Preconditions, abort conditions and recovery checks run automatically in the dry run.
Agents have read-only deployment access and cannot dispatch the workflow.
Automatic triggering is limited to services whose contract allows it, through a deterministic alert rule and a separate production-auto environment that only the alerting app can reach.
A rehearsal within your policy window recorded actor, approver, times, time to recover and lessons.
CI blocks a release when the last rehearsal is stale, except for an incident-ID deploy that opens a rehearsal ticket.
The incident commander declares recovery.

Where to go next after the rollback pipeline

Q20 · Production approval gate The environment, identities and OIDC trust the rollback job reuses.

Q22 · Anomaly detection The deterministic alert rules that may trigger a pre-approved rollback.

Progressive delivery Flags and canaries that stop most bad releases before a rollback is needed.

Incident response Agents that correlate alerts, investigate the deploy and draft the post-mortem around a rollback.

Scoped AI incident triage The read-only incident agent that feeds the decision record.

Official references: GitHub deployment environments, GitHub MCP server, and the Deploy stage of the method.

Frequently asked questions

How does a rehearsed rollback pipeline keep production authority with humans?

A versioned rollback contract defines the target release, preconditions, and recovery checks, and a protected deploy job with short-lived credentials executes it when triggered by a named human or a pre-approved deterministic alert rule.

What role do coding agents play during a production rollback?

Coding agents can draft the rollback decision record and verify post-rollback recovery checks using read-only evidence, but they never hold production credentials or trigger the rollback themselves.