AI-Powered Disaster Recovery

Disaster recovery with AI coding agents covers two blast radii: a change an agent shipped, undone through checkpoints, revertable commits, feature flags and targeted reverts; and the infrastructure underneath, restored through point-in-time-recovery runbooks, backups and rehearsed failover. Both depend on recovery objectives stated in advance, drills that measure them, and a human at every irreversible step.

This guide is for the developer who carries the pager and the tech lead who owns a service’s recovery plan. The scenario it prepares you for: an agent refactors the auth module across 47 files, tests pass, you merge, and two hours later the admin API stops validating session tokens while three other PRs have landed on top. A month later the primary region goes dark mid-deploy, and the Postgres runbook points at a host that was decommissioned in the spring.

What you’ll walk away with from this disaster-recovery playbook

• A recovery-objective table (RTO and RPO per tier) that turns every later artifact into something you can test
• Per-tool settings and two prompts that keep an agent’s large change revertable one commit at a time
• A revert-first procedure for a merged agent change that is hurting production, with the prompt that drives it
• A prompt that generates a Postgres point-in-time-recovery (PITR) runbook from your real pgBackRest config, and a scheduled, read-only agent job that fails CI when the runbook drifts from that config
• A Chaos Mesh failover drill with pass and fail criteria measured against your RTO and RPO, and a named person who signs off
• A human-gated failover prompt, a blameless post-mortem prompt, and the failure modes that silently break recovery plans

Which disaster are you recovering from?

Name the blast radius before you pick a tool. The two kinds need different mechanics, but the same discipline.

	An agent’s change	The infrastructure
Blast radius	A commit, a PR or a deploy	A database, a cluster or a region, and everything on top of it
Recovery mechanism	Checkpoint, git revert, feature-flag flip, minimal hotfix	Restore from backup, replica promotion, regional failover
Measured against	Time from alert to rollback	RTO (time to restore service) and RPO (data you can afford to lose)
What decides success	Whether the change was small enough to undo cleanly	Whether the procedure was rehearsed against the real topology
Who approves the irreversible step	The on-call engineer who owns the service	The incident commander

The agent drafts, checks and explains at both scales. A person approves every step that cannot be undone: a force-push, a data restore, a replica promotion, a DNS cutover.

Set recovery objectives before you generate anything

Every artifact below depends on two numbers per service tier. A runbook that targets “fast recovery” cannot fail a drill; one that targets “RTO 15 minutes, RPO 5 minutes for payments” can. Start from this template and replace the values with the ones your business signs off:

Tier	Example services	RTO	RPO	Backup approach
Critical	auth, payments	15 min	0–5 min	Streaming replica plus continuous WAL archiving
High	core API	1 hour	5 min	WAL archiving plus frequent base backups
Medium	secondary features	4 hours	1 hour	Hourly snapshots
Low	internal tooling	24 hours	24 hours	Daily backups

Commit the table next to your runbooks, for example as ops/recovery-objectives.md, so every prompt below can point the agent at it instead of restating numbers.

Which model to use for recovery planning

Start on your tool’s default model: Claude Opus 5.5 in Claude Code (the default from v2.1.280, the latest channel) and GPT-6 Astra in Codex. Raise effort before you switch models. Pick Claude Fable 5.1 deliberately (/model fable) only for a tangled multi-region failover plan where your own attempts at higher effort fall short. Prices and the full lineup live on the models hub.

Make an agent’s change recoverable before it merges

The cheapest recovery is the one you set up before the agent starts. Each tool keeps a local undo for agent edits, but none of them replaces Git: a checkpoint lives on one machine, misses some kinds of change, and never covers what already merged.

Claude Code

Claude Code checkpoints its file edits before each prompt you send. Run /rewind, or press Esc twice with an empty prompt, and choose to restore the code, the conversation or both. Checkpoints do not track files changed by Bash commands such as rm or a migration script, so commit before you let the agent run anything destructive. Old checkpoints are cleaned up automatically, so do not treat them as an archive.

Put the durable rules in CLAUDE.md. From v2.1.277 on the latest channel, Claude Code also reads AGENTS.md when a project has no CLAUDE.md; on Bedrock, Google Cloud, Foundry and LLM gateways that needs v2.1.281. The rule text is in the block below this tab set.

Codex

In codex-cli 0.157.1, codex features list reports the undo feature as removed, so Git is your rollback mechanism. Run each large change on its own branch or worktree. A Codex cloud task produces a diff you inspect before it touches your checkout (codex cloud is marked experimental in 0.157.1):

# Terminal: review a cloud task's result, then apply it locally only if it is right
codex cloud list
codex cloud diff TASK_ID
codex cloud apply TASK_ID

TASK_ID comes from codex cloud list. Put the rules in AGENTS.md.

Cursor

Cursor keeps its own session undo for Agent edits (the Cursor CLI lists a /rewind command, checked 2026-08-28). Treat it as a session undo only, and confirm the controls in Cursor’s checkpoints documentation before you rely on them. Put the rules in a project rule under Rules, and let Bugbot review the pull request before a human looks at it.

The safety rules themselves are identical across the three tools:

## Change safety
- Work on a branch named agent/<task>. Never commit to main. Never force-push.
- Before a change that touches more than 10 files, list the files and wait for approval.
- Commit after each step that passes `npm test` and `npm run typecheck`. One logical step per commit.
- Never delete a file, edit a migration that has run, or change *.config.*, .env*, Dockerfile
  or infra/ without showing the diff and waiting for approval.
- If a test fails after a step, revert that step's commit and report why. Do not fix forward.

Two habits make the difference between a two-minute revert and an afternoon of untangling: commit in steps that each pass the tests, and pin down current behaviour in tests before the rewrite starts.

Copy-paste prompt: refactor in revertable commits

Refactor UserService from a monolithic class to the repository pattern. It touches about
30 files. Do not make the change in one pass. Follow this plan:

Commit 1: add the UserRepository interface and a Postgres implementation. Change nothing else.
Commit 2: inject the repository into UserService; UserService still uses its own methods.
Commit 3 onward: migrate one method per commit, starting with findById, then create.

Every commit must pass `npm test` and `npm run typecheck`, be revertable on its own with
`git revert`, and touch the minimum number of files. After each commit, print the commit
hash, the files changed and the test result, then stop and wait for me to say "continue".

Copy-paste prompt: behaviour-capture tests before a refactor

Before we refactor src/middleware/auth/, write behaviour-capture tests in
src/middleware/auth/auth-middleware.behavior-capture.test.ts.

For each exported middleware:
1. Call it with a valid request and assert the exact status, headers and next() call.
2. Call it with an expired token, a missing token and a token for a disabled user, and
   assert the exact error response for each.
3. Assert the registration order of the middleware chain on /api/admin/* routes.

Do not change any production code. Run the new tests against the current code and show
that they pass. Then make one deliberate mutation (skip the token-expiry check), show
that at least one test fails, and revert the mutation.

The mutation step is how you know the tests can fail; a behaviour-capture suite that stays green when you break the code protects nothing. The full technique is in characterization tests.

Roll back an agent’s change that already shipped

Speed beats elegance once a merged change is hurting production. Work in this order and stop at the first step that restores service:

1. Flip the flag. If the change shipped behind a feature flag, turn it off. This is why agent-written changes should ship behind flags and canaries; the setup is in progressive delivery.

2. Revert the merge. Revert the merge commit on main and let CI deploy it:

   # Terminal: find the merge commit for the PR, then revert it (keep the main-line parent)
   git log --merges --oneline -20
   git revert -m 1 MERGE_SHA

   MERGE_SHA is the merge commit from the log. If the PR was squash-merged, find it with git log --oneline and run git revert SHA without -m 1. If later PRs depend on the agent’s code and the revert conflicts, skip to step 3 rather than untangling mid-incident.

3. Ship a minimal hotfix. Restore the behaviour that broke, and change nothing else; open the proper fix as a normal PR after the incident.

4. Check for deleted files. An agent’s refactor can delete a file whose absence nobody notices until the code path runs. List what the PR removed with git diff --name-only --diff-filter=D MERGE_SHA^1 MERGE_SHA and restore a file with git checkout MERGE_SHA^ -- PATH.

Copy-paste prompt: diagnose a production regression from a merged agent PR

Production incident. The error rate on /api/admin/* rose to 15% after PR #847 (an
agent-written refactor of the auth module) was merged and deployed.

Do not edit any files yet. Do this and report back:
1. Show exactly what PR #847 changed in the auth middleware pipeline.
2. Compare the middleware registration order on /api/admin/* before and after the merge.
3. List every removed function call, listener, error branch and config value in the diff.
4. Tell me whether `git revert -m 1` of the merge commit applies cleanly on current main,
   and which later commits touch the same files.
5. If a revert is not clean, propose the smallest hotfix that restores the old behaviour,
   as a diff, plus a test that fails on current main and passes with the hotfix.

Do not refactor, rename or improve anything. Restore behaviour only.

The regression test in step 5 is the verification: it fails on the broken build and passes on the fix, so the reviewer checks one test and one small diff instead of rereading the refactor.

Recover data an agent’s change corrupted

This is where the two blast radii meet: the code fix is a revert, but the data fix is a restore.

1. Stop writes from the bad code. Deploy the rollback or flip the flag first. Do not fix forward while bad writes continue.

2. Measure the damage with read-only queries. Ask the agent for SELECT statements that find rows written or updated during the incident window, and run them yourself against a read replica. Record the row counts before you change anything.

3. Restore into a side database. Use the PITR runbook below to restore to a timestamp just before the bad deploy, into a separate instance, never over the live primary.

4. Repair with a reviewed script. Have the agent write the repair script that copies correct values from the side database, run it first on a copy, compare row counts and checksums against step 2, and get a second person to approve it before it runs on production.

5. Close the gap. Add a database constraint or a validation test that would have rejected the bad writes, and record the failure class (see the failure taxonomy).

Generate a Postgres PITR runbook your on-call can follow

A point-in-time-recovery runbook is the recovery artifact on-call engineers reach for first, so write one they can follow half asleep. The pattern that works is continuous WAL archiving plus an external backup tool such as pgBackRest or WAL-G. A restore sets a recovery_target_time and needs an unbroken WAL chain from the base backup to that target. Have the agent write the runbook against your real configuration, not a template:

Copy-paste prompt: generate a Postgres PITR runbook

Read infra/pgbackrest.conf, infra/postgresql.conf and ops/recovery-objectives.md.
Write ops/runbooks/pitr.md: a point-in-time-recovery runbook for our Postgres 16 primary,
backed up with pgBackRest to S3 (stanza prod-main). Targets: RTO 15 min, RPO 5 min.

The runbook must:
1. Use `pgbackrest --stanza=prod-main info` to find the latest backup that finished
   before a given target timestamp.
2. Run `pgbackrest --stanza=prod-main verify` and stop, with a clear message, if any WAL
   segment between that backup and the target is missing.
3. Restore into a NEW data directory on a separate host with --type=time, --target and
   --target-action=promote. Never restore over the running primary.
4. Verify after the restore: row counts on orders, payments and users compared with the
   last known-good report, and the recovery end time in the Postgres log.
5. Be a numbered checklist. Mark each step REVERSIBLE or IRREVERSIBLE, and put
   "REQUIRES APPROVAL FROM THE INCIDENT COMMANDER" before the step that points traffic
   at the restored database.

Use only options that appear in `pgbackrest help restore` and the Postgres 16 docs. After
the runbook, list every option you used and the command that documents it.

The last instruction exists because the two things an agent most often gets wrong here are invented pgBackRest options and a skipped WAL-gap check. Run pgbackrest help restore and check each listed option before the runbook is merged. The core of a correct runbook looks like this:

# On the recovery host, as the postgres user
pgbackrest --stanza=prod-main info                      # backups and the WAL range each one needs
pgbackrest --stanza=prod-main verify                    # checks backup and WAL archive integrity
pgbackrest --stanza=prod-main --type=time \
  "--target=2026-09-25 14:02:00+00" --target-action=promote restore

Green backups can still hide a WAL gap

The classic silent failure: archive_command reports success while a lifecycle rule on the archive bucket expires older WAL segments. The restore then stops partway, mid-incident. Make the runbook check continuity with verify, and compare the bucket’s lifecycle rules with your retention setting.

Keep the runbook true with a scheduled drift check

A runbook rots the day someone renames a stanza or moves a host. Run a read-only agent against it on a schedule, from the default branch, and fail the job when it drifts from the config.

Claude Code

# Scheduled CI job on the default branch; ANTHROPIC_API_KEY comes from the secret store
claude -p "Read infra/pgbackrest.conf and ops/runbooks/pitr.md. Check that every stanza,
host, bucket and path in the runbook exists in the config. Print one line per mismatch.
End with DRIFT if there is at least one mismatch, otherwise CLEAN." \
  --allowedTools "Read,Grep,Glob" --permission-mode dontAsk > drift.txt
tail -n 1 drift.txt | tr -d '*[:space:]' | grep -qx CLEAN

--permission-mode dontAsk denies every tool that is not on the allow list, so the job can only read. The last line turns the verdict into the job’s exit code; tr -d strips Markdown such as **CLEAN** that the model may wrap around the word, so formatting cannot fail the job.

Codex

# Scheduled CI job on the default branch; CODEX_API_KEY comes from the secret store
codex -a never exec --sandbox read-only \
  "Read infra/pgbackrest.conf and ops/runbooks/pitr.md. Check that every stanza, host,
bucket and path in the runbook exists in the config. Print one line per mismatch. End with
DRIFT if there is at least one mismatch, otherwise CLEAN." -o drift.txt
tail -n 1 drift.txt | grep -qx CLEAN

-a goes before exec (in 0.157.1, codex exec does not accept it), and -o writes the agent’s last message to a file. The runner has to support the read-only sandbox: the Linux sandbox can fail inside an unprivileged container, so test the job on the runner image you schedule it on. If the model may format its verdict (for example **CLEAN**), strip the markup before the check with tail -n 1 drift.txt | tr -d '*[:space:]' | grep -qx CLEAN, or ask for the bare word CLEAN on its own final line.

Cursor

Create a scheduled Automation for a Cloud Agent with the same prompt, and have it open a pull request that fixes the runbook when it finds drift. A human reviews that pull request like any other; the agent never edits main.

Run the job with a token that can read the repository and nothing else (permissions: contents: read in GitHub Actions, and persist-credentials: false on checkout). The details of running agents safely in CI are in headless agents in CI.

Audit real backup settings instead of guessing them

A generated backup policy is only plausible. Connected to your cloud account through an MCP server with a read-only identity, the agent can read the actual bucket lifecycle rules, snapshot schedules and replica regions, and tell you where the RPO gap is. The AWS server is registered the same way in each tool; this example uses a profile bound to a read-only IAM role, and adds the proxy’s --read-only flag, which hides tools that may need write permissions, as a second layer on top of the role. The endpoint stays us-east-1; the region you query is passed as AWS_REGION metadata (without --metadata, it comes from the profile). In Claude Code or Codex, run /mcp afterwards; in Cursor, open the MCP settings. Confirm the tools you need are still listed. The managed AWS MCP Server is labelled preview in the awslabs README (checked 2026-09-26); in Claude Code you can also install it as the aws-core plugin from claude-plugins-official:

Claude Code

claude mcp add aws-mcp -- uvx mcp-proxy-for-aws-cli@latest \
  https://aws-mcp.us-east-1.api.aws/mcp --profile agent-readonly --metadata AWS_REGION=eu-west-1 --read-only

Codex

codex mcp add aws-mcp -- uvx mcp-proxy-for-aws-cli@latest \
  https://aws-mcp.us-east-1.api.aws/mcp --profile agent-readonly --metadata AWS_REGION=eu-west-1 --read-only

Cursor

Add this to ~/.cursor/mcp.json:

{
  "mcpServers": {
    "aws-mcp": {
      "command": "uvx",
      "args": ["mcp-proxy-for-aws-cli@latest", "https://aws-mcp.us-east-1.api.aws/mcp",
               "--profile", "agent-readonly", "--metadata", "AWS_REGION=eu-west-1", "--read-only"]
    }
  }
}

Azure, Google Cloud and the other providers, with their read-only settings, are covered in cloud platform MCP servers; Kubernetes and Terraform are in infrastructure MCP servers.

Copy-paste prompt: find the RPO gaps in your real backups

Using the aws-mcp tools, read-only: for the RDS instances and S3 buckets tagged
tier=critical or tier=high, list the automated backup retention, whether point-in-time
recovery is enabled, the cross-region replication target, and every lifecycle rule that
expires or transitions objects in the WAL archive bucket.

Compare each against ops/recovery-objectives.md. Output a table with one row per resource:
resource, tier, target RPO, what the current settings can actually deliver, and the gap.
Do not change any setting. For each gap, propose the Terraform change as a diff for
infra/, and say which AWS API call you read each fact from.

For Kubernetes workloads, Velero backs up cluster resources and persistent volumes. Ask for a Schedule covering the payments and auth namespaces whose cron interval meets the tier’s RPO, with a TTL that meets your retention policy, then prove it by restoring into a scratch namespace with velero restore create --from-backup BACKUP_NAME --namespace-mappings payments:payments-drill. A backup you have never restored is not a backup.

Prove failover works with a scheduled chaos drill

An untested recovery plan is a hypothesis. A scheduled Chaos Mesh experiment tests failover continuously: the Schedule resource (apiVersion: chaos-mesh.org/v1alpha1) runs a PodChaos pod-kill on a cron.

Copy-paste prompt: generate a weekly failover drill with pass and fail criteria

Write chaos/postgres-failover.yaml: a Chaos Mesh Schedule (apiVersion chaos-mesh.org/v1alpha1)
that runs every Saturday at 03:00 ("0 3 * * 6") in the staging cluster, with type PodChaos,
action pod-kill, mode one, selecting pods labelled app=postgresql-primary in namespace db,
concurrencyPolicy Forbid and historyLimit 5.

Then write ops/runbooks/failover-drill.md that defines pass and fail:
- recovery time = first successful write to the new primary minus the kill time; fail above 15 min
- data loss = replication lag on the standby at kill time; fail above 5 min
- a script, scripts/drill-report.sh, that pulls both numbers from our Prometheus
  (pg_replication_lag_seconds and the app's write-success counter) and prints PASS or FAIL

Use only fields defined in Chaos Mesh's Schedule and PodChaos types. List them at the end.

The manifest is the easy half. The pass and fail definition is what turns killing pods into evidence.

1. Run the drill in a staging cluster that mirrors production topology, never against customer traffic.

2. Let drill-report.sh record kill time, promotion time and first successful write automatically; nobody reads logs by hand.

3. Treat a FAIL as a P2 ticket with an owner, not a curiosity.

4. Have the service’s tech lead sign off each month’s drill results in the ticket. That signature, not a green backup dashboard, is your evidence that the RTO and RPO hold.

5. Feed each failed drill into the post-mortem prompt at the end of this guide.

Run the live failover with a human at every irreversible step

Mid-incident, use the agent to generate and check the next action, never to execute irreversible commands from free text. Keep the session read-only and let a person run each command. The wider incident loop (paging, triage, status updates) is in incident response.

Copy-paste prompt: plan a regional failover

We have lost us-east-1. Read ops/runbooks/pitr.md, ops/recovery-objectives.md and
infra/dns.tf. Do not run any command.

Produce an ordered checklist to fail payments, auth and the core API over to us-west-2.
For each step give: the exact command, whether it is reversible, what to check before
running it, and what output proves it worked. The first step must stop writes to the
failed primary. Before promoting a replica, check its replication lag against the RPO.
Mark every irreversible step (stopping writes, promotion, DNS cutover) as
REQUIRES INCIDENT COMMANDER APPROVAL.

The same discipline covers ransomware: the agent can read the backup catalogue and the file-modification timeline to find the last backup that finished before the first encryption event, and draft network-isolation rules, while a person runs the restore.

What breaks disaster recovery plans, and how do you recover?

• A WAL gap strands the restore. A lifecycle rule or a failing archive_command removed segments. Recovery: restore to the latest target the remaining chain supports, report the real data loss, and add verify plus a lifecycle check to the drift job.
• Replication lag exceeds the RPO at the worst moment. The write spike that precedes an outage leaves the standby minutes behind. Recovery: alert on lag against the RPO number, and make the failover checklist compare lag before promotion; see monitoring and observability.
• Split-brain after failover. The old primary still accepts writes after the standby is promoted. Recovery: fence the old primary first, snapshot it, export the rows it accepted after the promotion time (diff against the new primary) and replay them after review, then rebuild it as a standby with pg_rewind or a fresh base backup; keep “stop writes” as step one of every failover checklist.
• The drill passes and the real failover fails. Staging has three nodes and production has 30. Recovery: mirror production topology in the drill cluster, and rotate the injected failure (pod kill, network partition, disk full).
• The agent invents a flag or a CRD field. The runbook or manifest will not run. Recovery: the prompts above make the agent list every option with its source; check that list against --help or the CRD types before merging.
• A revert conflicts because later PRs build on the agent’s code. Recovery: ship the minimal hotfix instead of untangling commits during the incident, then schedule the proper fix. Revertable commits (above) make this rarer.
• The checkpoint you need is not there. The change came from a shell command that checkpoints do not track, or the snapshots were already swept. Recovery: Git is the durable record; commit per step so there is always a commit to return to.

Caution

When production is down after an agent-written merge, roll back first and diagnose second. Fixing forward during an active incident turns a two-minute revert into a long outage. The post-mortem asks which safety net was missing, never who approved the PR.

Turn the incident into a post-mortem and a regression test

The incident is not closed until the lesson is in the repository: a post-mortem, a regression test, and a changed runbook or rule. Feed the agent the real artifacts, not your memory of them.

Copy-paste prompt: draft a blameless post-mortem

Draft a blameless post-mortem from incident/timeline.csv, logs/incident-2026-09-25/ and
the drill-report.sh output for the last failed drill.

Structure: a one-paragraph summary; a minute-by-minute timeline; root cause, separating the
trigger from the underlying cause; customer impact (duration, failed requests, data loss
measured against the RPO); what went well; what went wrong; 3 to 5 action items, each with
an owner role, a priority and the test, alert or runbook change that proves it is done.
Describe systems and decisions, never individuals. Mark every statement the logs do not
support as UNVERIFIED.

Store the post-mortem next to the runbook it should change, so the next drill tests against the lesson from the last incident. For incidents caused by an agent’s change, tag the failure class and follow the organisational process in agent incidents.

Where to go next with disaster recovery

Incident response Paging, triage and status updates around the recovery steps on this page.

Progressive delivery Flags, canaries and SLO-triggered rollback so a bad agent change never reaches everyone.

Rehearsed rollback pipeline Make rollback a pipeline step you have practised, with human production authority.

Failure taxonomy Classify what went wrong and find the earliest control that catches it next time.

Monitoring and observability Replication-lag and backup-health alerts that catch recovery gaps early.

Infrastructure as code Keep failover infrastructure reproducible and reviewed.