AI-Powered Disaster Recovery

Disaster recovery for AI-assisted teams spans two blast radii: a change an agent shipped, recovered through checkpoints, incremental commits, feature flags, and targeted reverts; and the infrastructure under it, recovered through Postgres point-in-time-recovery runbooks, Velero backups, and rehearsed failover. Both share one discipline — stated recovery objectives, a rehearsed procedure, and a human at every irreversible gate.

The AI agent just refactored your authentication module across 47 files. Tests pass, types check, you merge. Two hours later, session tokens are not being validated in the admin API, three other PRs have merged on top, and rolling back is no longer a single click. A month later a different disaster arrives: your primary region goes dark mid-deploy, the status page is still green because nobody updated it, and the runbook is a Confluence page last edited fourteen months ago referencing a Postgres host decommissioned in the spring.

Different scales, same question — how fast can you get back, and how much do you lose on the way? Disaster recovery is the one area where “we will write the docs later” quietly becomes “we lost six hours and a chunk of customer data.” The good news: the slow, tedious parts — drafting precise runbooks, writing chaos experiments that actually assert your RPO, capturing the current behavior of code before an agent rewrites it, turning a messy timeline into a blameless post-mortem — are exactly what AI coding tools are good at, as long as you anchor them to real tools and keep a human at the approval gate.

What this disaster-recovery playbook gives you

Safety-net configuration for all three tools so an agent cannot make a 47-file change you cannot unwind
A prompt that forces an adversarial review of an agent’s diff, hunting the silent behavioral changes tests do not catch
Recovery procedures for AI-introduced regressions at every stage: pre-merge, post-merge, and data corruption
A copy-paste prompt that turns your stack into a Postgres point-in-time-recovery runbook with explicit RTO/RPO targets and WAL-chain verification
A Chaos Mesh Schedule that kills your primary DB pod every week and asserts failover under a 5-minute RPO — generated, not hand-typed from memory
A human-gated playbook for the live regional outage, and a prompt that drafts a blameless post-mortem from real logs
A list of failure modes that silently break recovery plans, and how to catch them before the disaster does

Two blast radii, one discipline

A recovery plan is only useful if it names what it recovers from. Almost everything that goes wrong on an AI-assisted team lands in one of two buckets, and they want different tools:

The change. An agent introduced a regression. The blast radius is a commit, a PR, or a deploy. Recovery is a checkpoint, a revert, a feature-flag flip, or a hotfix — measured in minutes, and mostly a question of whether the change was small enough to undo cleanly.
The infrastructure. A database, a region, or a cluster is gone. The blast radius is everything on top of it. Recovery is a restore or a failover — measured against objectives you set in advance, and mostly a question of whether you rehearsed.

The discipline is identical in both: state the objective before the incident, rehearse the procedure, keep a person at every irreversible step, and capture what you learned. The rest of this guide runs that discipline at both scales.

Recovery objectives come first

Every DR artifact downstream depends on two numbers per service tier: RTO (how long until service is restored) and RPO (how much data you can afford to lose). Write them down explicitly before you generate anything — a runbook that targets “fast recovery” is useless; one that targets “RTO 15 min, RPO 5 min for payments” is testable.

A realistic starting matrix looks like this:

Tier	Example services	RTO	RPO	Backup approach
Critical	auth, payments	15 min	0–5 min	Synchronous/streaming replica + WAL archiving
High	core API	1 hour	5 min	WAL archiving + frequent base backups
Medium	secondary features	4 hours	1 hour	Hourly snapshots
Low	internal tooling	24 hours	24 hours	Daily backups

Prevention: the safety net before the agent starts

The best disaster recovery is not needing it. Each tool has a native place to put the rules, and the rules are what keep a large agent-driven change revertable.

Cursor’s checkpoint system provides automatic rollback points:

Checkpoints are automatic snapshots of the Agent’s changes to your codebase
To roll back, click Restore Checkpoint on the relevant earlier request, or the + button when hovering over a message
Checkpoints are stored locally and are separate from Git — use Git for permanent version control, not checkpoints

Add explicit safety rules:

SAFETY REQUIREMENTS:
Before any multi-file refactoring:
1. List all files that will be modified
2. Verify the test suite passes BEFORE making changes
3. After changes, run the full test suite
4. If any test fails, revert ALL changes and report what went wrong

NEVER delete files without explicit user confirmation.
NEVER modify configuration files (*.config.*, .env*, Dockerfile) without showing the diff first.

Claude Code works with Git directly. Establish commit-based safety:

SAFETY PROTOCOL:
Before starting any multi-file modification:
1. Run: git stash (save any uncommitted work)
2. Create a safety branch: git checkout -b ai/[task-description]
3. Commit after each logical step with descriptive messages
4. Run tests after each commit
5. If tests fail, use git diff to identify the problem

After completing the task:
- Run the FULL test suite (npm test)
- Run type checking (npm run type-check)
- Run linting (npm run lint)
- Show the complete diff from main for review

NEVER force-push. NEVER modify the main branch directly.

Claude Code’s permission system provides an additional safety layer — file writes require explicit approval unless auto-approved in settings.

Codex cloud tasks run in isolated sandboxes with built-in safety:

SAFETY PROTOCOL:
- All changes happen in a new branch (never modify main)
- Cloud tasks cannot push directly to main
- Every task produces a PR for human review
- Worktrees provide isolation between parallel tasks

Before submitting a PR:
1. Run the full test suite
2. Run the linter
3. Generate a comprehensive PR description explaining all changes
4. Flag any files that were deleted or had configuration changes

Codex’s sandboxed environment means a runaway task cannot affect your local environment or other branches.

Review the diff adversarially

Type checks and a green suite are the weakest evidence you have about an agent’s refactor. Ask for the review that hunts what they miss:

I need you to review the changes the AI agent just made. Be adversarial.

1. Read the complete diff (git diff main...HEAD)
2. For each modified file, check:
   - Were any function calls removed that are not covered by tests?
   - Were any middleware registrations, event listeners, or side effects changed?
   - Were any error handling paths altered or removed?
   - Were any configuration values changed?
3. List files that were DELETED and verify nothing else depends on them
4. Check for behavioral changes that would pass type-checking but change runtime behavior
   (e.g., changing an async function to sync, changing error types, altering return values)

I am especially worried about silent behavioral changes that tests do not catch.

Make the change small enough to undo

Two habits do more for recoverability than any tooling: commit in steps that each pass tests, and capture existing behavior in tests before the rewrite.

I need to refactor the UserService from a monolithic class to the repository pattern.
This affects approximately 30 files.

DO NOT make all changes at once. Follow this incremental plan:

Commit 1: Create the UserRepository interface and implementation.
          Change nothing else. All existing code still works.
          Run tests.

Commit 2: Add the repository as a dependency to UserService.
          UserService still uses its internal methods (no behavior change).
          Run tests.

Commit 3: Migrate the first method (findById) to use the repository.
          Run tests.

Commit 4: Migrate the next method (create). Run tests.

Continue this pattern for each method. Every commit must:
- Pass all tests
- Be independently revertable
- Change the minimum number of files

Stop after each commit so I can review before continuing.

Before we refactor the auth middleware, I need behavior-capture tests.

For each middleware function in /src/middleware/auth/:
1. Write a test that calls the middleware with a valid request and asserts the EXACT current behavior
2. Write a test that calls it with an invalid request and asserts the EXACT error response
3. Write a test that verifies the middleware chain order (which middleware runs before which)

These tests should FAIL if the refactoring changes any observable behavior.
They are intentionally brittle - that is the point. They are our safety net.

Name them clearly: auth-middleware.behavior-capture.test.ts
After the refactoring is complete and verified, we will remove these in favor of proper tests.

Deploy so that a rollback is a flag, not a fire drill

Feature flags for AI-generated changes

Deploy AI-assisted changes behind feature flags. If something goes wrong, flip the flag instead of rolling back the deployment.
Canary deployments

Route 5% of traffic to the new version. Monitor error rates, latency, and key business metrics for 30 minutes before expanding.
Automated rollback triggers

Set up automatic rollback when error rate exceeds 2x baseline or p99 latency exceeds 3x baseline.
Post-deployment monitoring

Watch dashboards for 4 hours after deploying AI-generated changes. The failure modes of AI code are often subtle — edge cases and race conditions rather than crashes.

Recovering from a change you shipped

The agent broke tests before merge

The easiest recovery, and the one where the tool you used decides the mechanics.

Use Cursor’s checkpoints to restore the last good state:

Find the request just before the breaking change
Click its Restore Checkpoint button (or the + button when hovering over that message) to return to that state
Alternatively, use Cmd+Z to undo recent edits

Checkpoints are stored locally and separate from Git, so reach for Git once you need a durable rollback point.

# If working on a branch (recommended):
git diff main  # See what changed
git stash       # Save current state
git checkout main  # Return to clean state

# If you committed incrementally (recommended):
git log --oneline -10  # Find the last good commit
git revert HEAD~3..HEAD  # Revert the bad commits

It merged, and production is hurting

Speed matters more than elegance here. Point the agent at the specific PR and forbid it from improving anything on the way past:

We have a production incident. The error rate on /api/admin/* endpoints spiked to 15%
after merging PR #847 (AI-assisted auth module refactoring).

Investigate immediately:
1. Show me exactly what PR #847 changed in the auth middleware pipeline
2. Compare the middleware registration order before and after the PR
3. Check if any route-specific middleware was removed or reordered
4. Look at the error logs - what specific error are users hitting?
5. Identify the minimal fix (do not refactor further - just restore correct behavior)
6. Generate the fix as a hotfix PR with only the necessary changes

Speed is critical. Do not optimize or clean up. Fix the regression and nothing else.

It corrupted data

The most dangerous scenario, and the one where the change-level and infrastructure-level playbooks meet: the fix is a restore.

Stop the bleeding

Deploy the rollback immediately. Do not try to fix forward when data integrity is at risk.
Assess the damage

Query the database for records modified during the incident window. Determine the scope of corruption.
Restore from backup

Use the point-in-time recovery runbook below to restore affected data to the state before the incident.
Root cause analysis

Identify which AI-generated code caused the corruption. Was it a missing validation? A wrong query? A race condition?
Prevent recurrence

Add specific test cases for the failure mode. Add database constraints that would catch the corruption at the data layer. Update your agent rules to prevent similar patterns.

Recovering the infrastructure underneath

The single most valuable DR artifact for most teams is a Postgres point-in-time-recovery runbook that an on-call engineer can follow at 3 AM without thinking. The pattern that actually works in production is WAL archiving plus an external base-backup tool — pgBackRest or WAL-G — not a server-side plpgsql function. PITR restores set restore_command and recovery_target_time and rely on an unbroken WAL chain between the base backup and the target time.

Have your AI tool write the runbook against your actual config rather than a generic template:

Write a point-in-time-recovery runbook for our Postgres 16 primary,
backed up with pgBackRest to S3 (stanza name: prod-main).

Targets: RTO 15 minutes, RPO 5 minutes.

The runbook must:
1. List exact pgBackRest commands to find the latest full backup whose
   stop time is at or before a given target timestamp.
2. Verify the WAL chain is intact from that base backup to the target
   time, and explicitly fail closed if there is a gap.
3. Show the recovery.signal + postgresql.auto.conf settings
   (restore_command, recovery_target_time, recovery_target_action=promote).
4. Include a post-restore verification step: row counts on the three
   highest-value tables and a check that the latest archived WAL segment
   is replayed.
5. Be a numbered checklist a sleep-deprived on-call can follow, with the
   one irreversible step (promote) called out and gated behind explicit
   confirmation.

Output as Markdown. Do not invent flags — use only real pgBackRest and
Postgres options.

Read what it produces critically. The two things AI most often gets wrong here are inventing pgBackRest flags and glossing over the WAL-chain gap check — the exact failure that strands you mid-recovery. If a command looks unfamiliar, check it against pgbackrest --help before it goes in the runbook.

Where each tool fits

The runbook itself is identical regardless of tool — it is a Markdown file in your repo. What differs is how you drive the generation and keep it honest over time.

Open the runbook in the editor and use agent mode so Cursor can read your actual pgbackrest.conf, docker-compose.yml, and migration files for ground truth instead of guessing host names and stanza names. Iterate inline: when a step looks wrong, select it and ask Cursor to fix just that step. Use a checkpoint before letting it touch multiple files so you can roll back a bad rewrite in one click.

This is the fastest loop when DR config lives in the same repo you are editing and you want to see diffs as they happen.

Use headless mode to regenerate and validate the runbook as part of a scheduled DR drill or a pre-release check, so it never silently rots:

claude -p "Read infra/pgbackrest.conf and ops/runbooks/pitr.md. \
Verify every pgBackRest flag in the runbook exists in this version, \
and that the stanza name matches the config. List any drift as a \
checklist of fixes. Exit non-zero if the runbook references a host or \
stanza not present in the config." \
  --allowedTools "Read,Grep"

Wire that into a weekly CI job. A red build means your runbook drifted from reality — which is exactly when you want to find out, not during the outage. The --allowedTools "Read,Grep" flag keeps the check read-only so it can run unattended.

Hand the whole task to Codex Cloud: give it the repo and a task like “regenerate ops/runbooks/pitr.md from the current pgBackRest config and open a PR.” Running on GPT-5.6 Sol, it works in an isolated environment against a checked-out copy, so it can grep the real config, regenerate the runbook, and push a branch for review without touching your machine.

For local work, codex in a worktree keeps the DR changes isolated from your feature branches, and its GitHub integration can open the PR for a human to approve.

Backing up the rest of the stack

A database is rarely the whole story. For Kubernetes-hosted workloads, Velero backs up cluster resources and persistent volumes and is the standard tool to prompt your AI assistant to configure — not an invented internal backup service.

Generate a Velero Schedule manifest that backs up the "payments" and
"auth" namespaces every 6 hours, snapshots their PersistentVolumes,
excludes Secrets of type kubernetes.io/service-account-token, and sets
a 30-day TTL. Then give me the exact `velero backup describe` and
`velero restore create` commands to validate a restore into a scratch
namespace. Use only real Velero CRD fields and CLI flags.

For object storage and managed databases, prefer the provider’s native cross-region replication and point-in-time features (for example, RDS automated backups with PITR, or S3 Cross-Region Replication) over rolling your own. Ask your AI tool to generate the Terraform for those, then review the IaC the same way you reviewed the runbook.

Testing it: chaos drills that assert your RPO

An untested DR plan is a hypothesis. The cheapest way to test failover continuously is a scheduled Chaos Mesh experiment that kills your primary DB pod and checks that the system recovers inside your stated objectives. The Schedule CRD (apiVersion: chaos-mesh.org/v1alpha1) runs a PodChaos pod-kill on a cron.

Write a Chaos Mesh Schedule (apiVersion chaos-mesh.org/v1alpha1) that:
- runs every Saturday at 03:00 (cron "0 3 * * 6")
- uses type "PodChaos" with action pod-kill, mode one
- targets pods with label app=postgresql-primary
- sets concurrencyPolicy Forbid and historyLimit 5

Then write a follow-up runbook (Markdown) describing how I assert the
drill passed: how to confirm the standby was promoted, how to measure
actual recovery time against a 15-minute RTO, and how to measure
replication lag at kill time against a 5-minute RPO. Use only real
Chaos Mesh CRD fields.

The generated manifest is the easy half. The runbook that defines pass/fail against your RTO/RPO is the half that makes the drill meaningful — without it, you are just killing pods and hoping. Treat a missed RTO in a Saturday drill as a P2 ticket, not a curiosity.

Run the drill in a staging cluster that mirrors production topology, never against live customer traffic.
Capture the timeline automatically: kill time, promotion time, first successful write to the new primary.
Compare measured RTO/RPO to your targets and file a ticket for any miss.
Feed the drill’s logs straight into the post-mortem prompt below so the gaps turn into action items instead of being forgotten by Monday.

When the real thing happens: the human-gated playbook

Mid-incident, AI is best used to generate and validate the next action — never to fire irreversible commands directly from free text. Keep a person at the approval gate for anything that promotes a replica, reroutes DNS, or disables writes.

We have lost us-east-1. Read ops/runbooks/pitr.md and infra/dns.tf.

Produce an ordered recovery checklist to fail payments, auth, and the
core API over to us-west-2, where each step states: the exact command,
whether it is reversible, and what to verify before moving on. Flag
every irreversible step (replica promotion, DNS cutover, disabling
writes in the failed region) as REQUIRES HUMAN APPROVAL. Do not run
anything — output the checklist only.

For a ransomware scenario the same discipline applies: use the tool to find the last clean backup before encryption and to draft network-isolation rules, then have a human execute. A useful, specific Claude Code invocation:

claude -p "Given the pgBackRest backup catalog in this repo's logs/ \
directory and the file-modification timeline in incident/timeline.csv, \
identify the most recent backup whose stop time precedes the first \
encryption event, and explain how you ruled out later backups." \
  --allowedTools "Read,Grep"

Note the claude -p headless form and the read-only tool allowlist: it analyzes and recommends, it does not restore.

When recovery plans break

Both scales fail in predictable ways. Watch for these:

WAL chain gaps. Bucket lifecycle rules or a silently-failing archive_command expire segments between your base backup and target time. The restore aborts partway. Fix: the runbook must verify continuity, not assume it.
Replication lag exceeds RPO at the worst moment. Under the write spike that often precedes an outage, your standby falls minutes behind. Promoting it loses more data than your RPO allows. Fix: alert on lag against the RPO number, and have the failover step check lag before promoting.
Failover that strands writes. You promote the standby but the old primary is still accepting writes (split-brain), or in-flight writes never replicated. Fix: the playbook’s first step is always disable writes on the failed primary, before promotion.
The drill passes, the real thing fails. Staging has 3 nodes; production has 30 and a different topology. Fix: run drills against a cluster that mirrors production scale, and rotate the failure injected.
AI invents flags or fields. A generated pgBackRest flag or Chaos Mesh field that does not exist makes the artifact non-runnable. Fix: always validate generated commands against --help or the CRD reference before committing — treat AI output as a draft, never as gospel.
You cannot roll back because other changes depend on the agent’s code. This is what incremental commits buy you. If one commit introduced the issue, revert that commit; if the changes are tangled, a targeted hotfix beats an untangling operation during an incident.
The agent deleted files nobody noticed for weeks. Git has your back: git log --diff-filter=D finds deleted files and git checkout <commit>^ -- <filepath> restores them. Add a CI check that flags deletions for extra scrutiny in review.
The backup strategy does not cover AI-specific failure modes. Database backups and code in Git cover most of them. The unique risk is a subtle behavioral change that passes every check — which is why behavior-capture tests on critical paths and feature-flagged deploys belong in the backup strategy, not just in the coding standard.

Closing the loop: the post-mortem

The incident is not over until the learning is captured. AI is genuinely good at turning a messy timeline and a wall of logs into a structured, blameless post-mortem — provided you feed it the real artifacts.

Draft a blameless post-mortem from the attached incident timeline and
logs (incident/timeline.csv, logs/incident-2026-06-05/).

Structure: one-paragraph executive summary, a minute-by-minute timeline,
root cause (distinguish trigger from underlying cause), customer impact
(duration, requests failed, estimated data loss), what went well, what
went wrong, and 3-5 concrete action items each with an owner role and a
priority. Keep it blameless: describe systems and decisions, never
individuals. Flag any place where the logs are insufficient to determine
what happened.

Run this with Claude Fable 5 (/model fable) when the causal chain is tangled across services — the reasoning quality is worth the cost on the one document everyone will read; fall back to Opus 5 if budget is a concern. Keep the output in the repo next to the runbook it should improve, so the next drill tests against the lessons from the last incident.

Where to go next with disaster recovery

Incident Response On-call automation and AI-assisted incident handling.

CI/CD Pipelines Build the deployment safety nets this playbook depends on.

Security Compliance Security-focused recovery planning, the ransomware angle, and audit trails.

Performance Testing Load-test the DR systems before you rely on them.

Monitoring and Observability The replication-lag and backup-health alerts that catch DR gaps early.

Infrastructure as Code Keep the failover infrastructure reproducible and reviewed.