Fault injection testing: prove timeouts, retries, and circuit breakers fire

Fault injection testing, also called error injection, deliberately breaks a dependency (adds latency, resets connections, returns 500s, kills pods) to prove that timeouts, retries, and circuit breakers fire before production meets the same failure. Coding agents write these experiments quickly. The proof comes from a human-approved fault matrix and from tests that go red when the protection is removed.

Your order service passes every test, then hangs for 40 seconds the first time PostgreSQL slows down under load. The connection pool fills, requests queue, and the timeout everyone assumed existed turns out to be a server-side setting that network latency never trips. An agent wrote the retry wrapper last sprint, and nobody read it closely, because nobody reads every line anymore.

This page is for developers who own a service’s resilience code and tech leads who sign off on staging chaos experiments. It assumes you already have integration tests against a real database; fault injection stresses the same seams.

What you get from agent-built fault injection tests

• A fault matrix format that turns “the service must be resilient” into rows an agent can implement and a reviewer can approve in two minutes
• A working Toxiproxy suite for a Node.js and PostgreSQL service, including the control test that proves the proxy is really in the path
• A circuit-breaker test for opossum that counts upstream calls instead of trusting nock.isDone()
• A bounded Chaos Mesh workflow for staging, with server-side validation and an abort command
• Four copy-paste prompts, a CI workflow with least-privilege permissions, and recovery steps for the ways chaos experiments go wrong

Where fault injection fits among your tests

Fault injection runs at two layers. Start at the application layer, because it is deterministic and cheap enough for every pull request, and move to the system layer only after each service fails fast on its own.

Layer	Tools	Runs where	Proves	Cadence
Application	Toxiproxy (TCP proxy), nock or MSW (HTTP mocks)	Laptop and CI	One service’s timeouts, retries, fallbacks, and pool hygiene	Every pull request
System	Chaos Mesh, LitmusChaos (Kubernetes CRDs)	Staging cluster	Recovery, blast radius, and steady-state SLOs across services	On demand or scheduled, behind approval

Load behavior is a different question. If you need to know how many requests per second the service handles before it degrades, see load, stress, and benchmark testing first; chaos against an idle staging cluster proves little.

Write the fault matrix before the agent writes code

Agents are good at producing resilience tests and bad at choosing the numbers. Ask one to “add timeout tests” and it invents a 5-second bound that matches nothing in your config, then writes code to satisfy it. So the human decision comes first, as a table in docs/resilience/order-service.md:

Dependency	Fault	Expected behavior	Bound
PostgreSQL	4 s latency on every response	createOrder rejects with the client-side query timeout; no pooled client leaks	Fails within 2.5 s
PostgreSQL	Connection reset	One retry with backoff, then a 503	Two attempts in total
Inventory API	Error rate above 50% (opossum compares strictly) over at least volumeThreshold calls (here 5)	Breaker opens; next call serves the cached fallback with no network call	Zero upstream calls while open
Inventory API	3 s response time	Counts as a failure (breaker timeout 1 s)	Fails within 1.2 s

Each row is an acceptance criterion in Given/When/Then shape, the same contract described in executable acceptance criteria. You review and approve this table instead of the test code. The agent’s job is to make each row a test that fails today if the protection is missing.

Inject dependency faults locally with Toxiproxy

Toxiproxy is a TCP proxy with an HTTP admin API on port 8474. Your app connects to the proxy instead of the database, and each test adds a “toxic” (latency, timeout, bandwidth, reset_peer, slicer, limit_data, slow_close) for its duration.

1. Start Toxiproxy next to your local PostgreSQL and create a proxy that listens on 5433 and forwards to 5432. Run this in a terminal:

   docker run -d --name toxiproxy --network host ghcr.io/shopify/toxiproxy:2.12.0
   curl -sf -X POST http://localhost:8474/proxies \
     -d '{"name":"postgres","listen":"127.0.0.1:5433","upstream":"127.0.0.1:5432"}'

   --network host is the simplest on Linux. On Docker Desktop (macOS or Windows), publish ports 8474 and 5433 and use host.docker.internal:5432 as the upstream, unless you have enabled Docker Desktop’s host networking.

2. Install the client: npm install --save-dev toxiproxy-node-client (4.1.0 on 2026-09-26).

3. Point the test pool at port 5433 and set a client-side query timeout. This is the gotcha the opening scenario hit: PostgreSQL’s statement_timeout measures execution time on the server, so latency injected on the network never trips it. Only query_timeout in node-postgres, which the client enforces, catches a slow network.

4. Write the suite. Reset Toxiproxy after every test so a failed assertion never leaves the proxy poisoned for the next one:

   test/resilience/db-faults.test.js

   import { Toxiproxy } from 'toxiproxy-node-client';
   import pg from 'pg';
   import { createOrder } from '../../src/services/order.service.js';
   
   const toxiproxy = new Toxiproxy('http://localhost:8474');
   const pool = new pg.Pool({
     host: '127.0.0.1',
     port: 5433, // through the proxy, never 5432
     database: 'app',
     user: 'postgres', // the CI service has only this role
     query_timeout: 2000, // client-side: trips on network latency
     connectionTimeoutMillis: 1000,
   });
   
   afterEach(() => toxiproxy.reset()); // removes every toxic, re-enables proxies
   afterAll(() => pool.end());
   
   test('control: a reset connection breaks the query, so the proxy is in the path', async () => {
     const proxy = await toxiproxy.get('postgres');
     await proxy.addToxic({ name: 'reset', type: 'reset_peer', stream: 'downstream', toxicity: 1, attributes: { timeout: 0 } });
     await expect(pool.query('SELECT 1')).rejects.toThrow();
   });
   
   test('createOrder fails fast when PostgreSQL responses are 4 s late', async () => {
     const warm = await pool.connect(); // the pool is empty after the control test
     warm.release(); // so open one now, or the connect timeout fires instead
     const proxy = await toxiproxy.get('postgres');
     await proxy.addToxic({ name: 'slow-db', type: 'latency', stream: 'downstream', toxicity: 1, attributes: { latency: 4000, jitter: 0 } });
   
     const start = performance.now();
     await expect(createOrder(pool, { sku: 'ABC-1', qty: 1 })).rejects.toThrow('Query read timeout');
     expect(performance.now() - start).toBeLessThan(2500);
     expect(pool.totalCount - pool.idleCount).toBe(0); // no client left checked out
   });

The examples use ES modules, so they assume Jest configured for ESM or TypeScript (ts-jest or babel-jest). Vitest runs them if you set test.globals: true (or import test, expect, afterEach and afterAll from vitest) and swap npx jest for npx vitest run.

The control test matters more than it looks. If someone points the pool back at 5432, every latency test still “passes” because the query returns quickly. The control test goes red instead.

The latency test warms the pool for a reason. After the control test the pool holds no live client, so without the warm-up the 4 s latency delays the PostgreSQL handshake. Then connectionTimeoutMillis fires first with timeout exceeded when trying to connect, and a loose /timeout/i assertion stays green even with query_timeout deleted. The exact Query read timeout message (node-postgres) rules that out. If you also want to prove the connect timeout, give it its own test that adds the latency toxic before any client exists and asserts timeout exceeded when trying to connect.

Now hand the rest of the matrix to your agent. The prompt is the same in every tool:

Copy-paste prompt: Toxiproxy suite from the fault matrix

Read docs/resilience/order-service.md (the fault matrix), src/services/order.service.ts
and src/db/pool.ts.

Toxiproxy is already running: admin API on http://localhost:8474, proxy "postgres"
listening on 127.0.0.1:5433 and forwarding to PostgreSQL on 5432.

Write test/resilience/db-faults.test.js with toxiproxy-node-client:
- one test per PostgreSQL row of the matrix, using the bound from the matrix, not a
  number you choose
- a control test that adds a reset_peer toxic and expects the query to fail
- toxiproxy.reset() in afterEach
- assert the specific error (message or class), elapsed time, and that no pooled
  client leaks; never a bare rejects.toThrow() on a fault test

Run: npx jest test/resilience/db-faults.test.js
If a test fails because a protection is missing (for example there is no client-side
query_timeout), add it to src/db/pool.ts and re-run until green. Do not change a bound
in the matrix or a test's expected value to make it pass; stop and tell me instead.
Finish with a table: matrix row, test name, protection it exercises, file and line
of that protection.

The tools differ in who starts the containers and how the result comes back to you.

Claude Code

Run claude in the repository and paste the prompt. Claude Code iterates on its own: generate, run, read the failure, add the missing timeout, run again. Start Toxiproxy yourself first (step 1) so the agent does not need Docker permissions. Save the prompt as a project skill in .claude/skills/resilience-suite/SKILL.md if you will run it for every service.

Codex

Paste the prompt into the Codex CLI (codex) or the IDE extension. Start Toxiproxy and PostgreSQL before the session. With the default approval policy (on-request), Codex asks before running a command its sandbox does not allow, so expect to approve npx jest. If the sandbox refuses the connection to Toxiproxy on localhost, run the test file yourself and paste the failure output back into the session.

Codex’s extra lever here is --output-schema on codex exec, which makes the final summary machine-readable for a pull request comment. codex exec cannot stop to ask for approval. The workspace-write sandbox also blocks network by default, so the -c override below opens it for this run. Without it, npx jest cannot reach Toxiproxy, and the agent may read the refused connection as a missing protection. The override opens all outbound network for the run, not only localhost, so use it only on a machine without production credentials. Permission profiles are the newer mechanism (beta in Codex CLI), but they do not compose with --sandbox, so pick one. Save the prompt above to prompts/resilience.md, then run this in a terminal:

codex exec --sandbox workspace-write \
  -c sandbox_workspace_write.network_access=true \
  --output-schema schemas/resilience-summary.json \
  -o resilience-summary.json "$(cat prompts/resilience.md)"

schemas/resilience-summary.json is a JSON Schema you write once: an object with a rows array of {row, test, protection, location} (the root must be an object), so CI can check that every matrix row maps to a test.

Cursor

Give Cursor’s agent the prompt with the files @-mentioned (@docs/resilience/order-service.md @src/db/pool.ts), so it reads your real pool settings. Cursor runs the test command in its integrated terminal; start Toxiproxy (step 1) before you send the prompt. Check the proposed diff to pool.ts before you accept it: that file is the only production change this task should make.

Prove the circuit breaker opens without calling the network

The common circuit-breaker test mocks the API to fail ten times, trips the breaker, and asserts nock.isDone() === false. That assertion proves nothing: with ten interceptors and six calls, isDone() is false whether or not the breaker blocked the sixth request. Count the upstream calls instead.

This test assumes the client exports its opossum breaker. opossum opens on an error percentage (errorThresholdPercentage, default 50) once volumeThreshold requests have been seen, not after a fixed count of consecutive failures, so the test reads the threshold from the breaker:

test/resilience/inventory-breaker.test.js

import nock from 'nock';
import { getInventory, inventoryBreaker } from '../../src/clients/inventory.client.js';

afterEach(() => nock.cleanAll());

test('breaker opens, then serves the fallback with zero upstream calls', async () => {
  let upstreamCalls = 0;
  nock('https://inventory.internal')
    .get(/.*/)
    .times(50)
    .reply(() => {
      upstreamCalls += 1;
      return [500, { error: 'boom' }];
    });

  const opened = new Promise((resolve) => inventoryBreaker.once('open', resolve));
  const volume = Math.max(inventoryBreaker.options.volumeThreshold ?? 1, 1);
  for (let i = 0; i < volume; i += 1) {
    await getInventory('SKU-1').catch(() => {}); // the fallback may resolve these
  }
  await opened;

  const callsWhenOpened = upstreamCalls;
  const result = await getInventory('SKU-1');
  expect(result.source).toBe('fallback-cache');
  expect(upstreamCalls).toBe(callsWhenOpened); // the breaker, not luck, stopped the call
});

Copy-paste prompt: circuit breaker state machine tests

Read src/clients/inventory.client.ts. It wraps the inventory API in an opossum
CircuitBreaker exported as inventoryBreaker, with a fallback that returns
{ source: 'fallback-cache', ... }.

Write test/resilience/inventory-breaker.test.js with nock and Jest:
1. Open: fail enough calls to cross volumeThreshold and errorThresholdPercentage
   (read both from inventoryBreaker.options; do not hardcode). Count upstream calls
   with a counter inside nock's reply function and assert the next call returns the
   fallback with no new upstream call. Do not use nock.isDone() as evidence.
2. Half-open: use jest.useFakeTimers() (under ESM, import { jest } from
   '@jest/globals'; in Vitest use vi.useFakeTimers()) to advance past
   options.resetTimeout, assert
   the 'halfOpen' event, allow exactly one probe; on success expect 'close', on
   failure expect 'open' again.
3. Slow responses: nock .delay() longer than options.timeout must emit 'timeout'
   and count as a failure.
Assert on emitted events ('open', 'halfOpen', 'close', 'timeout', 'fallback'),
never on private fields. Run the file and fix only the tests, not the client.

Run system-level chaos against staging with Chaos Mesh

Once each service fails fast, check that the system recovers. Chaos Mesh (v2.8.4 on 2026-09-26) expresses every experiment as a Kubernetes custom resource, so the agent’s output is a manifest you can diff, validate, and bound. One network delay on the database, limited to the staging namespace and 10 minutes:

chaos/postgres-latency.yaml

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: postgres-latency
  namespace: staging
spec:
  action: delay
  mode: all
  selector:
    namespaces: [staging]
    labelSelectors:
      app: postgresql
  delay:
    latency: '200ms'
    jitter: '50ms'
  duration: '10m'

A staged run chains several faults in a Workflow resource with templateType: Serial, inserting Suspend templates as recovery gaps and a deadline on each step. Validate every manifest against the CRD schemas installed in the cluster without creating anything:

kubectl apply --dry-run=server -n staging -f chaos/

A client-side dry run (--dry-run=client) never reaches the API server. Chaos Mesh’s validating webhooks and server-side field validation never run, so it can pass manifests the cluster would reject. Ask the agent for the staged workflow and the runbook together, so the abort command is written before anyone applies anything:

Copy-paste prompt: staged Chaos Mesh workflow and runbook

Read k8s/staging/ to learn our namespaces, labels and service names. Use only
labels that appear there.

Create chaos/staged.yaml: a Chaos Mesh Workflow (apiVersion chaos-mesh.org/v1alpha1)
with an entry template of templateType Serial and these children, each followed by a
Suspend template with deadline 5m as the recovery gap:
1. NetworkChaos delay 200ms, jitter 50ms, on app=postgresql, deadline 10m
2. PodChaos pod-kill, mode fixed-percent, value "50", on app=payment-service
   (pod-kill is instantaneous; the deadline only bounds the step)
3. StressChaos memory stressor, 2 workers, size 256MB, on app=order-service, deadline 8m
4. NetworkChaos partition, direction both, from tier=backend to app=postgresql, deadline 3m
Every selector must include namespaces: [staging].

Write chaos/RUNBOOK.md: the steady-state metrics to capture right before the run
(p99 latency and error rate per service), the abort thresholds (p99 above 2 s or
error rate above 1% for 2 minutes), the abort command
(kubectl delete workflows.chaos-mesh.org staged -n staging), and how to confirm recovery.

Validate with: kubectl apply --dry-run=server -n staging -f chaos/
Fix schema errors until it passes. Do not run kubectl apply without --dry-run.

Claude Code

Paste the prompt in a session that has kubectl access to staging; the dry run needs it.

To let Claude Code watch pods and events while an experiment runs, add the Kubernetes MCP server with a read-only config (read_only = true in the TOML file):

claude mcp add kubernetes -- npx -y kubernetes-mcp-server@latest --config ~/.config/k8s-mcp.toml

Codex

Chaos manifests need no running cluster to write, so this is a good Codex cloud task (codex cloud is experimental in Codex CLI 0.157.1). Save the prompt to prompts/staged-chaos.md without its last two lines (the dry run needs cluster access), then submit it from the terminal:

codex cloud exec --env ENV_ID "$(cat prompts/staged-chaos.md)"
codex cloud diff TASK_ID    # review the manifests
codex cloud apply TASK_ID   # bring them into your working tree

ENV_ID is your Codex cloud environment (browse them with codex cloud), and TASK_ID is printed by the first command. Run the server-side dry run locally after apply. To give the local CLI read-only cluster access, add the same MCP server with the same read-only TOML file: codex mcp add kubernetes -- npx -y kubernetes-mcp-server@latest --config ~/.config/k8s-mcp.toml.

Cursor

Use the same prompt with @k8s/staging/ mentioned so Cursor’s agent reads your real labels. For cluster visibility, add the server to .cursor/mcp.json:

{ "mcpServers": { "kubernetes": { "command": "npx", "args": ["-y", "kubernetes-mcp-server@latest", "--config", "/path/to/k8s-mcp.toml"] } } }

Caution

Keep the MCP server read-only and its kubeconfig limited to the staging context. An agent that can both apply chaos and read the dashboards can also “fix” a failing experiment by deleting it early and reporting success. The apply and abort decisions stay with a human or with CI.

How do you know a fault test can fail?

A green resilience suite written by an agent is only evidence if each test would go red without the protection it claims to exercise. Check that mechanically, not by reading the tests:

• Negative control per protection. Remove each timeout, retry, or breaker in turn and run the suite; the matching test must fail. This is a targeted form of mutation testing, and oracle strength explains how to automate it with a mutation tool on changed files.
• An observable effect in every test. Every fault test asserts elapsed time, a specific error, or a call count. A bare rejects.toThrow() passes on a typo.
• The control test stays in the suite. It proves the traffic goes through the proxy.
• The tests are outside the agent’s edit scope. Put test/resilience/ and docs/resilience/ under CODEOWNERS so a later agent cannot relax a bound to get green; see protecting the oracle.

Copy-paste prompt: prove each resilience test can fail

Start from a clean working tree (git status shows no changes).
For each test in test/resilience/, find the production protection it claims to
exercise (timeout, retry, breaker, fallback, pool setting). Then, one at a time:
1. disable that protection with the smallest possible edit
2. run only the matching test file
3. record whether the test failed, and with what message
4. revert the edit with git checkout -- <file> before the next one
Report a table: test, protection, file:line, red without it (yes/no).
Any "no" is a vacuous test: propose a stronger assertion, but do not apply it.
Leave the working tree clean at the end (git status must show no changes).

Who signs off. The service owner approves the fault matrix and the negative-control table in the pull request, which belongs in the evidence bundle. A tech lead or the on-call SRE approves each staging chaos run through the CI environment gate below, and records the steady-state baseline taken right before it.

Run fault tests in CI and gate staging chaos

Application-layer tests run on every pull request. Staging chaos runs only on manual dispatch, after the tests pass, behind a GitHub environment whose required reviewers approve before the job receives the staging credentials.

.github/workflows/fault-injection.yml

name: Fault injection
on:
  pull_request:
  workflow_dispatch:
    inputs:
      experiment:
        description: Chaos Mesh experiment to apply to staging
        type: choice
        options: [postgres-latency, staged]

permissions:
  contents: read

jobs:
  resilience-tests:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_DB: app
          POSTGRES_HOST_AUTH_METHOD: trust # throwaway CI database only
        options: >-
          --health-cmd pg_isready --health-interval 5s --health-retries 10
      toxiproxy:
        image: ghcr.io/shopify/toxiproxy:2.12.0
        ports: ['8474:8474', '5433:5433']
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm ci
      - name: Route PostgreSQL through Toxiproxy
        run: >-
          curl -sf -X POST http://localhost:8474/proxies
          -d '{"name":"postgres","listen":"0.0.0.0:5433","upstream":"postgres:5432"}'
      - run: npx jest test/resilience/

  staging-chaos:
    if: github.event_name == 'workflow_dispatch'
    needs: resilience-tests
    runs-on: ubuntu-latest
    environment: staging # required reviewers approve before secrets are exposed
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - name: Write the staging-only kubeconfig
        env:
          KUBECONFIG_DATA: ${{ secrets.STAGING_CHAOS_KUBECONFIG }}
        run: printf '%s' "$KUBECONFIG_DATA" > "$RUNNER_TEMP/kubeconfig"
      - name: Abort if staging is already degraded
        env:
          KUBECONFIG: ${{ runner.temp }}/kubeconfig
        run: ./scripts/chaos-preflight.sh
      - name: Apply the experiment (spec.duration or deadlines bound it)
        env:
          KUBECONFIG: ${{ runner.temp }}/kubeconfig
          EXPERIMENT: ${{ inputs.experiment }}
        run: kubectl apply -n staging -f "chaos/${EXPERIMENT}.yaml"

Three details carry the safety. The input is a choice and reaches the shell through an environment variable, never interpolated into the command. The kubeconfig belongs to a service account whose RBAC allows only chaos-mesh.org resources in the staging namespace. And scripts/chaos-preflight.sh is yours to write: it should exit non-zero when an alert is firing or an incident is open, so chaos never lands on an already unhealthy system. A dispatch runs the workflow and that script from whichever branch the dispatcher picks, so limit the staging environment’s deployment branches to the default branch; otherwise a feature branch can swap in its own preflight script and run it with the staging kubeconfig once a reviewer approves.

When fault injection experiments go wrong

Failure modes and recovery

The experiment outlives its window. A chaos resource whose controller restarts, or a manifest without duration, can keep injecting faults. Recovery: kubectl delete networkchaos postgres-latency -n staging (or kubectl delete workflows.chaos-mesh.org staged -n staging), then confirm with kubectl get networkchaos,podchaos,stresschaos -n staging that nothing is left. Prevention: reject any manifest without duration or a deadline in review.

The blast radius escaped staging. A selector without namespaces matches every namespace Chaos Mesh is allowed to touch. Recovery: delete the resource, then check which pods restarted with kubectl get events -A --field-selector reason=Killing. Prevention: the staging-only service account in CI, and Chaos Mesh’s own namespace filter (install with --set controllerManager.enableFilterNamespace=true and annotate only staging with chaos-mesh.org/inject=enabled), so production namespaces are never eligible.

The latency test passes but the timeout never fired. The pool is connected to 5432, or only statement_timeout is set. Recovery: the control test catches the first case; a client-side query_timeout fixes the second. Assert the exact Query read timeout message so the test cannot be satisfied by the connect timeout.

The agent weakened the test to get green. The bound went from 2.5 s to 5 s, or rejects.toThrow(/timeout/) became rejects.toThrow(). Recovery: revert the test file, and restate in the prompt that bounds come from the matrix only. Prevention: CODEOWNERS on test/resilience/.

The baseline was stale. Comparing against last week’s dashboard hides a regression. Recovery: rerun with a baseline captured in the 10 minutes before injection, as chaos/RUNBOOK.md requires.

The retry storm is worse than the outage. Three layers that each make three attempts turn one slow call into 27 requests at the dependency. Recovery: add a fault-matrix row that counts attempts at the dependency (Toxiproxy plus a request counter), and keep retries at one layer only.

Where to go next with fault injection

How strong is your oracle? Automate the negative controls with mutation testing on changed code.

Protecting the oracle Keep the fault matrix and resilience tests out of the agent's edit scope.

Progressive delivery for agent-written changes Canaries and automatic rollback: the production-side answer to the faults you rehearsed.

Incident response Turn each chaos finding into a runbook entry and each real incident into a new matrix row.

Infrastructure MCP servers Kubernetes and other cluster servers, with install commands and read-only setups.

Disaster recovery The region-scale failures that chaos in one staging cluster does not cover.