Skip to content
Developer Toolkit Developer Toolkit Developer Toolkit

Error Injection and Fault Injection Testing

Your payment service degrades silently when PostgreSQL gets slow under load. The connection pool fills, requests queue, timeouts you assumed existed never fire, and the first signal you get is angry customer tweets twenty minutes after the latency spike started. The happy-path tests all pass. The bug only shows up when a dependency misbehaves — which is exactly the condition you never test.

Error injection (also called fault injection) fixes this by deliberately breaking dependencies in a controlled way: slow down a database query, kill a downstream pod, drop packets between two services, or make an external API return 500s on demand. Chaos engineering is the systems-level extension of the same idea — running those faults against a live (usually staging) system to verify it holds. This guide shows how to use AI to generate the experiments, the runbooks, and the resilience code, against a concrete stack rather than hand-writing YAML from memory.

What You’ll Walk Away With

Section titled “What You’ll Walk Away With”
  • A repeatable workflow for injecting latency, dependency failures, and resource pressure with Cursor, Claude Code, and Codex
  • Real, runnable artifacts: a Toxiproxy setup for local fault injection and a Chaos Mesh manifest for Kubernetes
  • Copy-paste prompts that name your stack and produce experiments plus rollback-aware runbooks, not generic advice
  • A test that proves your timeouts, retries, and circuit breakers actually fire under failure
  • A “When This Breaks” checklist for the failure modes of chaos experiments themselves

The Two Layers of Fault Injection

Section titled “The Two Layers of Fault Injection”

Fault injection happens at two layers, and you want AI help at both.

Application / local injection

Inject faults in-process or at a local proxy (Toxiproxy, nock, fault-injecting HTTP clients). Fast, deterministic, runs in CI. Best for proving a single service’s timeout, retry, and fallback logic.

System / chaos injection

Inject faults at the infrastructure layer (Chaos Mesh, LitmusChaos) against a running cluster: pod kills, network delay, partitions, resource stress. Best for verifying recovery, blast radius, and steady-state behavior across services.

Start at the application layer because it is cheap and deterministic, then graduate to system-level chaos once individual services are proven resilient.

Workflow 1: Inject Dependency Latency Locally with Toxiproxy

Section titled “Workflow 1: Inject Dependency Latency Locally with Toxiproxy”

The fastest way to prove a service handles a slow dependency is to put a controllable proxy in front of it. Toxiproxy sits between your app and PostgreSQL (or Redis, or any TCP service) and lets you add latency, cut bandwidth, or sever the connection on demand.

A minimal real setup for a Node/Express service talking to PostgreSQL:

test/resilience/db-latency.test.js
import { Toxiproxy } from 'toxiproxy-node-client';
import { Pool } from 'pg';


const toxiproxy = new Toxiproxy('http://localhost:8474');


// App connects to PG *through* the proxy on 5433, not directly on 5432.
const pool = new Pool({ host: 'localhost', port: 5433, database: 'app' });


test('order creation fails fast when the DB is slow, instead of hanging', async () => {
  const proxy = await toxiproxy.get('postgres');
  // Inject 4s of latency on every query through the proxy.
  await proxy.addToxic({ type: 'latency', attributes: { latency: 4000 } });


  const start = Date.now();
  // statement_timeout / pool timeout should trip well before 4s.
  await expect(createOrder(pool, { sku: 'ABC', qty: 1 })).rejects.toThrow(/timeout/i);
  expect(Date.now() - start).toBeLessThan(3000);


  await proxy.removeToxic('latency'); // always clean up
});

The point is not the proxy plumbing — it is that you now have a deterministic way to assert “this code fails fast.” Let AI generate the experiment matrix and the assertions for your real service.

Open the service under test plus its data-access layer, then use Agent mode so Cursor can see your real timeout configuration and pool settings.

Workflow 2: System-Level Chaos with Chaos Mesh

Section titled “Workflow 2: System-Level Chaos with Chaos Mesh”

Once individual services fail fast, verify the cluster recovers. Chaos Mesh is a Kubernetes-native chaos platform whose experiments are plain CRDs — which means AI can generate them accurately and you can review them like any other manifest. A single NetworkChaos that delays traffic to your database for ten minutes:

chaos/postgres-latency.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: postgres-latency
  namespace: staging
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - staging
    labelSelectors:
      app: postgresql
  delay:
    latency: '200ms'
    jitter: '50ms'
  duration: '10m'

Apply it with kubectl apply -f chaos/postgres-latency.yaml, watch your SLO dashboards, and remove it with kubectl delete -f (or let duration expire). Use AI to expand a single fault into a staged experiment with safety controls.

Reference your existing Kubernetes manifests so Cursor uses your real namespaces, labels, and service names instead of placeholders.

Verify the Resilience Patterns Fire

Section titled “Verify the Resilience Patterns Fire”

Injecting a fault is only half the test. The other half is asserting that your timeout, retry, and circuit-breaker logic actually engaged. A real circuit-breaker test using a fault-injecting mock for an external API:

test/resilience/circuit-breaker.test.js
import nock from 'nock';
import { getInventory } from '../../src/clients/inventory.client.js';


test('circuit opens after 5 consecutive failures, then serves the fallback', async () => {
  // Make the external API fail every call.
  nock('https://inventory.internal').get(/.*/).times(10).reply(500);


  // Drive enough failures to trip the breaker (threshold = 5).
  for (let i = 0; i < 5; i++) {
    await expect(getInventory('SKU-1')).rejects.toThrow();
  }


  // 6th call should short-circuit and return the cached fallback,
  // NOT hit the network again.
  const result = await getInventory('SKU-1');
  expect(result.source).toBe('fallback-cache');
  expect(nock.isDone()).toBe(false); // breaker prevented the 6th network call
});

CI/CD Integration: Continuous Fault Injection

Section titled “CI/CD Integration: Continuous Fault Injection”

Run the cheap application-layer fault tests on every PR, and schedule heavier chaos experiments against staging on a cadence. A scheduled workflow that runs the resilience suite and can also fire a single named chaos experiment on demand:

.github/workflows/continuous-chaos.yml
name: Continuous Fault Injection


on:
  schedule:
    - cron: '0 */6 * * *' # every 6 hours
  workflow_dispatch:
    inputs:
      experiment:
        description: 'Chaos Mesh manifest to apply (staging only)'
        required: false
        default: 'postgres-latency'


jobs:
  resilience-tests:
    runs-on: ubuntu-latest
    services:
      toxiproxy:
        image: ghcr.io/shopify/toxiproxy
        ports:
          - 8474:8474
          - 5433:5433
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - name: Run fault-injection resilience suite
        run: npx jest test/resilience/


  staging-chaos:
    if: github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - name: Pre-flight - abort if staging is unhealthy
        run: ./scripts/preflight.sh --slo-status green --active-incidents none
      - name: Apply chaos experiment (auto-expires via spec.duration)
        run: kubectl apply -n staging -f chaos/${{ github.event.inputs.experiment }}.yaml

When This Breaks

Section titled “When This Breaks”

Chaos experiments have their own failure modes. These are the ones that bite teams hardest.

Essential Tools and Resources

Section titled “Essential Tools and Resources”
Chaos Mesh Kubernetes-native chaos platform; experiments are plain CRDs, easy for AI to generate and for you to review.
LitmusChaos CNCF chaos engineering framework with a large experiment hub and GitOps-friendly workflows.
Toxiproxy Shopify's TCP proxy for deterministic latency, bandwidth, and connection-failure injection in tests and CI.
Principles of Chaos Engineering The foundational definition of steady state, hypothesis, blast radius, and minimizing harm.

Your Fault-Injection Journey

Section titled “Your Fault-Injection Journey”

  1. Map dependencies and define steady state. List every external call (DB, cache, queue, third-party API) and the SLO each must hold under stress.

  2. Inject at the application layer first. Use Toxiproxy and nock to prove each service’s timeouts, retries, and fallbacks fire deterministically in CI.

  3. Graduate to system-level chaos. Once services fail fast, run Chaos Mesh experiments against staging to verify recovery and blast radius.

  4. Codify safety. Bound every experiment with duration, scope selectors to staging, and keep abort commands in a runbook.

  5. Automate the cadence. Run cheap fault tests on every PR; schedule heavier chaos against staging and gate it behind pre-flight health checks.

What’s Next

Section titled “What’s Next”
Integration Testing Test the service interactions and database state that fault injection then stresses.
E2E Testing Verify user journeys survive the degraded states you inject.
Performance Testing Load and benchmark services so chaos experiments run against realistic traffic.
Security Testing Probe failure modes for the security regressions they can expose.

See what's new on the platform.

Privacy Policy Terms of Service