Skip to content

Rehearsed rollback with human production authority

A rollback path is ready only when it has a versioned runbook, a protected execution identity, observable recovery criteria, and recent rehearsal evidence. An agent may prepare the command, identify the target revision, gather impact data, and verify a dry run. Production execution remains with the declared approver and platform gate unless a separately reviewed low-risk policy explicitly permits automation.

Scorecard question: How quickly and safely can you reverse a harmful production change?

Maximum-score answer: A tested one-command or one-job rollback has scoped credentials, named approval, deterministic recovery checks, and a recorded rehearsal.

Record these fields beside the deployment workflow:

service: checkout-api
rollback_target: immutable release identifier
command: ./scripts/rollback --release RELEASE_ID
approver_role: incident-commander
executor_identity: protected-deploy-job
preconditions:
- current release matches EXPECTED_RELEASE
- rollback target is retained and signed
recovery_checks:
- error-rate burn alert clears for 10 minutes
- checkout synthetic succeeds from two regions
abort_conditions:
- data migration is not backward compatible
- rollback target has an active security block
evidence_retention: incident record and deployment audit

This is an illustrative repository contract. Replace the command and criteria with the deployment platform’s tested interface.

  1. A deterministic alert or human report creates an incident record.
  2. The diagnostic identity gathers read-only evidence and proposes a target.
  3. CI validates target existence, compatibility, permissions, and dry-run checks.
  4. The named approver evaluates impact and authorizes the protected job.
  5. The deployment identity executes one immutable rollback operation.
  6. Monitoring evaluates recovery criteria; the incident commander declares recovery.
  7. The incident generates regression evidence and the next intent.md.

A rollback is not automatically safe. Database migrations, queues, external side effects, cache formats, and mobile clients can make backward movement riskier than a forward fix.

Prepare a rollback decision record for the current incident.
Use only read-only deployment metadata and the versioned runbook.
Name the current and target releases, compatibility risks, missing evidence,
recovery queries, abort conditions, and required human approver.
Do not execute, approve, or access production credentials.
After the protected job completes, compare the recorded metrics
with the runbook's recovery criteria. Cite query IDs and timestamps.
Separate recovered, not recovered, and inconclusive signals.
Draft follow-up actions; do not declare the incident closed.

The target artifact is gone. Retain immutable releases and test retrieval during rehearsal.

Schema changes are not backward compatible. Use expand/contract migrations, forward recovery, or an explicitly tested data restoration plan.

The command works but recovery is unknown. Define service and user-facing checks before the incident.

Rehearsal uses different permissions. Exercise the real protected job in a safe environment and audit the same identity path.

  • The runbook names immutable current and target revisions.
  • The execution identity is distinct from the coding agent.
  • Preconditions, abort conditions, and recovery checks are automated where possible.
  • A rehearsal records command, actor, time, result, and lessons.
  • Data migrations and external side effects have an explicit recovery path.
  • The incident commander—not the model—declares recovery.

Use Deploy for the protected release gate and Maintain for incident re-entry.