Skip to content
AI Developer Toolkit

Find your level — free Buy the Toolkit Buy Toolkit Buy the Toolkit Buy Toolkit Open the guides Read
Navigation menu
Buy the Toolkit Find your level — free Buy the Toolkit Open the guides Sign in

Recovery Patterns

PL

Level 2 How-to Updated

Recovery patterns for AI agents are recipes that have Claude Code, Codex or Cursor protect one flaky dependency with a maintained library (opossum, cockatiel, Resilience4j or tenacity) instead of a hand-written state machine: a circuit breaker, a retry that cannot amplify an outage, and an ordered fallback. A contract test that trips the breaker on purpose proves each one.

This page is for developers who own a service with a fragile downstream call and for tech leads who decide what a resilience pull request must prove. Your payment provider has a 90-second outage at 2 a.m. Every checkout request waits on the dead dependency, the connection pool drains, and the whole app falls over, not only payments. Asked to “add a circuit breaker”, an agent writes 200 lines of class CircuitBreaker that nobody tests. The recipes below give it a named library, a named call and a test to pass instead.

What you get from these recovery recipes

Section titled “What you get from these recovery recipes”
  • A resilience contract written as a test before the agent changes the call, so “the breaker opens” and “a 400 never retries” are pass or fail
  • Five copy-paste prompts: the contract test, an opossum circuit breaker, a cockatiel retry nested around a breaker, a read-path fallback, and a review of the finished diff
  • Exact commands to run the work with Claude Code and Codex, and the Cursor equivalent
  • A Sentry MCP loop that picks the call to protect from production evidence rather than a guess
  • The ways agent-written resilience code passes review and amplifies an outage, with a recovery for each

Which recovery pattern does this call need?

Section titled “Which recovery pattern does this call need?”

Decide per call, not per service. The pattern depends on whether the call is a read or a write and whether a repeat is safe.

CallPatternNode libraryCheck that proves it
Slow or dead dependency on the request pathCircuit breaker with a timeout and a fallbackopossum 10.0.0Breaker opens after N failures, fallback runs, half-open probe closes it
Idempotent read (GET, lookup by ID)Retry with jittered backoff, nested around a breakercockatiel 4.0.05xx retries, 4xx never retries, total calls capped
Non-idempotent write (POST /charge)Breaker and a queue; no blind retryopossum plus your job queueThe same idempotency key reaches the provider on every attempt
Read with a degraded answer availableOrdered fallback: primary, replica, cache, defaultPlain async functionsA 404 propagates; only infrastructure errors fall through

Versions are the npm latest tags checked on 2026-09-26. On the JVM, name Resilience4j and its Spring Boot @CircuitBreaker(name = ..., fallbackMethod = ...) annotation. On Python, name tenacity (9.1.4) for retries and pybreaker (1.4.1) for the breaker. Naming the library is what stops the agent from writing its own.

Write the resilience contract as a test first

Section titled “Write the resilience contract as a test first”

A circuit breaker you have never tripped in a test is a circuit breaker you debug during an incident. Write the contract first, watch it fail against the unprotected call, and only then let the agent change the code. The contract is the thing a reviewer reads instead of the diff.

The test drives the breaker through a factory that accepts option overrides. That design choice matters: the test passes resetTimeout: 50 and uses real timers, because fake timers can stall the async I/O of an HTTP stub and hang the test.

Cases 5 and 6 are the ones agents skip. An opossum timeout rejects the call and runs the fallback, but it does not cancel the request, so the provider can still complete a charge that your code has already sent to the fallback queue. opossum can abort through an AbortController, but in opossum 10.0.0 that controller is shared by the whole breaker: one timeout aborts every charge in flight, and autoRenewAbortController renews it only when the breaker half-opens or closes, so later calls start with an already-aborted signal. Case 6 catches that; a signal created per call passes it. Even an aborted request may have reached the provider, so the idempotency key is what makes the queued retry harmless.

Wrap the flaky call in a circuit breaker

Section titled “Wrap the flaky call in a circuit breaker”

With the contract red, the implementation is a small, bounded change. The prompt is the same for every tool; how you run it differs.

Terminal window
# Terminal, repository root. Flags checked against Claude Code 2.1.283; the
# v2.1.285 change is from the Claude Code changelog (checked 2026-10-12).
# claude -p starts in Manual mode (auto mode on third-party providers or with
# telemetry off, from v2.1.285); --permission-mode dontAsk denies anything that
# would prompt, on every provider. Reads and read-only commands still run.
claude -p "$(cat prompts/payment-breaker.txt)" \
  --permission-mode dontAsk \
  --allowedTools "Read" "Edit" "Write" "Bash(npm test *)"

Save the breaker prompt below as prompts/payment-breaker.txt. Install opossum yourself first, so the agent needs no network access and cannot add packages. The Bash(npm test *) rule does not limit what the tests run: npm test executes whatever package.json says, and Write can change that file. Review any change to package.json in the diff, or add --disallowedTools "Edit(package.json)" "Write(package.json)".

The result should be short. With opossum it is configuration, a filter and a fallback, with no state machine in sight:

import CircuitBreaker from 'opossum';


export function createPaymentBreaker({ charge, enqueue }: PaymentDeps, overrides = {}) {
  const breaker = new CircuitBreaker(
    // A signal per call: aborting it cancels this charge and no other
    (order: Order, key: string) =>
      charge(order, key, { signal: AbortSignal.timeout(2500) }),
    {
      name: 'payments.charge',
      timeout: 3000, // backstop above the per-call abort
      errorThresholdPercentage: 50,
      volumeThreshold: 20, // never open on a handful of requests
      resetTimeout: 30000,
      errorFilter: (err: HttpError) => err.status >= 400 && err.status < 500,
      ...overrides,
    },
  );
  // opossum calls the fallback with the same arguments, so the key is reused
  breaker.fallback(async (order: Order, key: string) => {
    await enqueue('payments', { order, idempotencyKey: key });
    return { status: 'queued' as const };
  });
  for (const state of ['open', 'halfOpen', 'close'] as const) {
    breaker.on(state, () => recordBreakerState(breaker.name, state));
  }
  return breaker;
}


// In the route: breaker.fire(order, order.idempotencyKey)

opossum 10.0.0 ships no TypeScript types, and @types/opossum is at 8.1.9 (both checked 2026-09-26). An option that is newer than the types, such as coalesce, fails type checking even though opossum reads it. Tell the agent to extend the type locally rather than casting the whole options object to any.

Add retries that cannot amplify an outage

Section titled “Add retries that cannot amplify an outage”

Naive retries multiply load during a partial outage: three retries on every client is a self-inflicted traffic spike on a dependency that is already struggling. Two rules prevent it. Jittered backoff stops clients from retrying in lockstep, and a strict predicate stops a 400 or a non-idempotent POST from being retried at all.

Nest the retry outside the breaker, so every retry counts toward the breaker’s failure rate and a sustained outage opens it. cockatiel does this in one line with wrap(retryPolicy, breakerPolicy). One detail breaks the combination: if the breaker has a fallback, the retry never sees a failure and never retries. Put the fallback outside both.

Leave the payment write out of this recipe. POST /charge gets a breaker and a queue, and the queue worker retries with the same idempotency key. If your payment provider does not accept an idempotency key, do not retry the charge automatically at all.

Degrade reads with an ordered fallback

Section titled “Degrade reads with an ordered fallback”

For reads, a fallback chain keeps the feature alive when the primary source is down: primary database, then read replica, then cache marked stale, then a safe default. Each tier must be labeled in the response, or you ship silent staleness. Keep it as an ordered list of async functions, not a FallbackChain<T> framework.

The rethrow rule separates resilience from a bug factory. A fallback that swallows a legitimate 404 and serves a cached profile is worse than the outage it was meant to hide.

Let production errors pick the call to protect

Section titled “Let production errors pick the call to protect”

Without production data, the agent guesses which call is flaky. Connect the Sentry MCP server and the agent reads the real issue (stack trace, event count, the failing span) before it proposes a breaker. Sentry’s server is remote, so there is nothing to install; only the registration differs between tools. Full read-only setup, token options and scopes are on Observability MCP servers.

Terminal window
# Checked against Claude Code 2.1.283
claude mcp add -s project --transport http sentry https://mcp.sentry.dev/mcp
claude mcp login sentry          # or run /mcp inside a session

On the OAuth screen, leave Triage Issues unchecked: the agent needs to read issues here, not resolve them. If you prefer a packaged skill to a bare server, Sentry’s plugin (sentry@claude-plugins-official, or codex plugin add sentry@openai-curated for Codex, checked against 0.157.1 help; requires a signed-in Codex) bundles the same server with its sentry-debug-issue skill. Install one or the other, not both, or the server registers twice.

Before the connection, “add a circuit breaker to checkout” produces a breaker around whatever call looks risky. After it:

How do you prove recovery code works without reading every line?

Section titled “How do you prove recovery code works without reading every line?”

A resilience pull request is small, but its failure mode is invisible until an incident. Review the evidence, not the diff:

  1. Contract test: red against the old code, green against the new. It covers opening, the 4xx filter, the timeout, the half-open probe, the idempotency key and a call that follows a timeout.
  2. Retry count test: asserts the exact number of calls for a 503, a 404 and an open circuit. A retry policy with no count assertion fails silently: it passes every other test while multiplying load.
  3. Greps with empty output: git grep -nE "class [A-Za-z]*(CircuitBreaker|Retry|Backoff)" src/ returns nothing, so no hand-written state machine slipped in next to the library.
  4. Metric and alert: circuit_breaker_state_change exists and an alert fires when a breaker stays open. Paste one real metric line from a local run into the pull request description.
  5. One staged fault: before the first production rollout, inject the failure in staging with a proxy such as Toxiproxy and watch the breaker open on the dashboard. The method is on error injection testing.

The author signs off on steps 1–4 in the pull request. The service owner or on-call lead signs off on step 5 and on the thresholds, because they own the recovery time the thresholds encode. To check a finished diff against the list, run the review prompt in a fresh session:

What breaks when an agent writes resilience code?

Section titled “What breaks when an agent writes resilience code?”
  • The breaker opens on a blip. opossum’s volumeThreshold defaults to 0, so with errorThresholdPercentage: 50 one failure out of one request is enough to open the circuit. Recovery: set volumeThreshold to a request count that a normal quiet minute exceeds, and base resetTimeout on the dependency’s observed recovery time (the Sentry prompt above), not on a round number.
  • Retries turn a brownout into an outage. A retry inside the breaker, or no breaker at all, lets every client keep hammering the dependency. Recovery: wrap(retry, breaker) with the retry outermost, a jittered backoff, and a test that asserts the total call count.
  • The retry never fires. A fallback registered on the breaker turns every failure into a success, so the retry around it has nothing to retry. Recovery: move the fallback outside the retry and breaker, and keep the retry count test.
  • The fallback masks a real bug. A fallback that catches everything converts 400s, auth failures and missing records into quiet wrong answers. Recovery: opossum’s errorFilter for 4xx, an explicit isRetryable predicate for cockatiel, and a test that a 404 propagates.
  • A timeout double-charges. The breaker times out, the fallback queues the order, and the original request completes at the provider anyway. Recovery: give each call its own AbortSignal.timeout() a little below opossum’s timeout, not the breaker’s shared getSignal(), and send the same idempotency key on the original call and the queued one.
  • The open breaker hides the outage. With the fallback working, the user-facing error rate looks healthy and nobody is paged while a dependency is fully down. Recovery: alert on the breaker-state metric and on the fallback counter, not only on 5xx rates. Routing that alert is covered in monitoring and logging patterns.
  • The agent guesses the library’s API. Option names differ between opossum, cockatiel and older forks, and types can lag behind the package. Recovery: the contract test is the arbiter; connect a documentation MCP server such as Context7 so the agent reads the docs for the installed version.

Where to go next with recovery patterns

Section titled “Where to go next with recovery patterns”

Did this guide help you?

One tap, no sign-in.

not at all a lot
All 577 guides Free resources: Compare tools Glossary By role

Written by

See what's new on the platform.

Privacy Policy Terms of Service Cookie settings llms.txt pricing.md RSS