Distributed Systems Development with AI

Distributed systems development with AI agents works contract-first: the OpenAPI or AsyncAPI contract changes before any code, and a machine check proves each change. Claude Code, Codex, and Cursor then build one saga step, one consumer, or one schema version at a time, each gated by a contract test, a compensation test, or a trace that survives every hop.

You add one field to the order-service API and three other services start returning 500s in staging. The trace stops after two hops because a Kafka consumer never propagated the trace context, and the payment saga skipped its refund. The agent you drove could see only the repository you had open, so it treated order-service as a standalone application.

This guide is for developers who own one service in a multi-repository system and for tech leads who have to make agent-written changes across service boundaries safe to merge. It assumes you already write specs before prompting; if not, start with spec-driven development.

What contract-first agent work gives you

• An integration section for CLAUDE.md, AGENTS.md, or a Cursor project rule that tells a single-repository agent which contracts it must obey
• A five-step loop for changing a service boundary, where each step ends in a check that fails loudly
• Copy-paste prompts for a contract-first endpoint, a saga step with its compensation, a trace-safe consumer, a dual-publish schema migration, and a cross-repository plan
• The CI gates that let you merge an agent’s cross-service change without reading every line: contract lint, contract tests, compensation tests, and a trace-continuity test
• Verified MCP servers (Sentry, Grafana, Kubernetes) that let the agent read live telemetry instead of guessing
• Recovery steps for broken traces, orphaned saga state, contract drift, and look-alike MCP packages

Why does an agent break services it cannot see?

Microservices split one system across repositories, languages, and teams. An agent loads one repository, so its default assumption, that this service stands alone, is wrong exactly where production breaks. The fix is to keep the contracts inside each repository (a contracts/ directory, a Git submodule, or a published package) and to name them in the file the agent always reads.

Claude Code

Add an integration section to CLAUDE.md. Claude Code also reads AGENTS.md when a repository has no CLAUDE.md (v2.1.277 and later on the latest channel), so one shared file can serve both Claude Code and Codex.

## Service boundaries (order-service)
Contracts live in contracts/ and are the source of truth:
- contracts/payment-api.yaml (OpenAPI 3.1): payment-service REST API
- contracts/inventory-events.yaml (AsyncAPI): events we consume
- contracts/order-events.yaml (AsyncAPI): events we publish

Rules:
- Never hand-write an HTTP client. Regenerate with `npm run generate-clients`.
- Change the contract first, in its own commit, and stop for review.
- Every consumer extracts trace context from message headers; every producer injects it.
- Every saga step ships with a compensation and a test that triggers it.
- Run `npm run contracts:check` and `npm test` before you say a task is done.

Codex

Put the same section in AGENTS.md at the repository root. Codex loads it at the start of every task in a trusted project (an untrusted project no longer supplies it, from 0.150.0). Name sibling repositories so a cross-service plan can say which ones need follow-up pull requests:

## Related services
- org/payment-service, org/inventory-service, org/notification-service
- Shared contracts: org/service-contracts (vendored into contracts/)
When a change touches a contract, list every downstream service that must change and in which order.

Cursor

Create a project rule that is always applied, with the same text as the Claude Code tab, and reference a specific contract in chat with @contracts/payment-api.yaml when the task is about that boundary. For rule file format and scoping, see Cursor project rules.

How do you change a service boundary with an agent?

Every boundary change follows the same loop. Each step ends with a check you run, not a diff you read.

1. Change the contract, and nothing else. The agent edits contracts/*.yaml in its own commit and stops. You review a short YAML diff instead of the handler code built on it.
2. Gate the contract. CI lints the spec and runs contract tests against every consumer, so a breaking change fails before anyone implements it.
3. Generate, then implement behind tests. Clients and server stubs come from the contract; the agent writes business logic until the failing tests pass.
4. Prove the runtime path. A trace-continuity test and, for sagas, a compensation test show the change works across the hop, not only inside the service.
5. Roll out in compatible steps. Producers and consumers deploy independently, so old and new formats must both be valid until a metric shows every consumer has moved.

Change the contract before the code

Copy-paste prompt: add an endpoint contract-first

Add a refund endpoint to payment-service, contract-first.
1. Read contracts/payment-api.yaml.
2. Add POST /payments/{paymentId}/refunds with:
   - request body: { amount: integer (minor units), reason: string, idempotencyKey: string }
   - 201: { refundId: string, status: "pending" | "completed", amount: integer }
   - 404 payment not found; 409 idempotency key already used with a different body;
     422 amount exceeds the refundable balance
3. Run `npm run contracts:check` and show me the output.
4. Stop. Do not generate handlers, clients or tests until I approve the contract.
After approval: regenerate the client with `npm run generate-clients`, write failing
tests for each response code, then implement the handler until they pass.

The stop in step 4 is the point of the prompt. A contract review is short and decides everything downstream; a handler review is long and decides little.

Gate contract drift in CI

Two checks catch most drift before it reaches staging: a spec linter on every contract change, and consumer-driven contract tests with Pact (@pact-foundation/pact on npm, 17.1.4 on 2026-09-26) that run in both the consumer’s and the provider’s pipeline. Wire them into one script so the agent and CI run the same command:

{
  "scripts": {
    "contracts:check": "spectral lint contracts/*-api.yaml && asyncapi validate contracts/order-events.yaml && npm run test:pact"
  }
}

spectral comes from @stoplight/spectral-cli and needs a .spectral.yaml in the repository root (extends: ["spectral:oas"] is the minimum); asyncapi comes from @asyncapi/cli. When a hand-edited client drifts anyway, ask for an audit that produces evidence rather than reassurance:

Copy-paste prompt: audit a client against its contract

Read contracts/payment-api.yaml and src/clients/payment-client.ts.
Produce a table with one row per operation and response code in the contract:
operation | response code | handled in client (file:line or "missing") | test that covers it (or "none").
Then list every retry, timeout and idempotency-key rule in the contract and whether the
client honours it. Do not change code. For each "missing" or "none", propose the smallest
failing test that would catch it.

How do you build a saga step you can trust?

A saga is where agent-written distributed code most often looks right and is wrong. The happy path works; the compensation is missing, runs twice, or has no timeout. Build one step at a time, and make the agent write the test that triggers the compensation before the code that performs it.

Copy-paste prompt: one saga step, its compensation, and a failing test

Implement one step of the orchestrated order saga: "reserve inventory".
Requirements:
1. Idempotent: calling it twice with the same orderId reserves once.
2. Compensation: releaseReservation(orderId), also idempotent.
3. Timeout budget of 30 s; on timeout the orchestrator runs the compensation.
Write the failing tests first:
- the next step (payment) throws -> the reservation is released exactly once
- the step is retried after a timeout -> stock is reserved once, not twice
Run `npm test -- saga` and show me the failing output. Then implement until green
and show the final test run. Do not add other saga steps. Do not weaken or delete
an assertion to make a test pass; if a test looks wrong, stop and tell me why.

How you keep that loop tight differs by tool:

Claude Code

Add a PostToolUse hook to .claude/settings.json so the saga suite runs after every edit and a regression in an earlier step surfaces in the same turn. Exit code 2 is what feeds the failure back to Claude; any other non-zero exit only shows the output to you:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [{ "type": "command", "command": "npm test -- saga --silent 1>&2 || exit 2" }]
      }
    ]
  }
}

For hook design and cost, see agent hooks.

Codex

Run the saga work in its own managed worktree, with the built-in :workspace permission profile (beta in Codex 0.157.1; write access to the checkout only). Save the saga prompt above as docs/saga-step.md first:

codex --worktree -c default_permissions=':workspace' \
  "Implement the reserve-inventory saga step per the prompt in docs/saga-step.md. Run npm test -- saga after each change."

Cursor

Paste the prompt into Agent in the service’s repository and let it run npm test -- saga in the terminal. Run each saga step in its own worktree (Cursor’s isolated Git checkout) so a bad compensation never mixes with a finished step. In the diff, look first at the test file: an agent that weakened an assertion is the failure to catch.

How do you keep trace context across async hops?

The recurring production bug in event-driven flows is a trace that dead-ends: a consumer reads a Kafka message but never extracts the trace context from its headers, so every downstream span starts a new trace. Make propagation a tested requirement, not a comment in the prompt.

Copy-paste prompt: a trace-safe Kafka consumer

Write the Kafka consumer for InventoryReserved in order-service using KafkaJS and
OpenTelemetry (@opentelemetry/api propagation).
1. Extract the W3C trace context from the message headers and make it the active
   context for the handler span.
2. Inject the current context into the headers of every message the handler produces.
3. Send poison messages to inventory-reserved.dlq after 3 attempts, keeping the headers.
Test: publish a message with a known traceparent, run the handler, and assert that
(a) the handler span has that traceId and (b) the produced message carries the same
traceId. Run the test and show the output.

Instrument one service end to end before templating it. Ask the agent to add OpenTelemetry auto-instrumentation (@opentelemetry/sdk-node with @opentelemetry/auto-instrumentations-node) to a single service, export over OTLP to a local Jaeger, send one request, and show the trace ID it expects you to find. Once that trace appears with the custom span, the same module becomes a shared package the other services import. The operations side, dashboards and alerting, belongs to monitoring and observability.

How do you change an event schema without breaking consumers?

A data format that crosses a service boundary changes through a sequence, not a commit. Both formats stay valid until a metric, not an assumption, shows every consumer has moved.

1. Add the new schema version next to the old one in the contract.
2. Deploy consumers that accept both versions.
3. Deploy the producer publishing both versions (or the new version with the old fields kept).
4. Watch a per-consumer metric of which version each service reads until the old one reaches zero.
5. Stop publishing the old version, then delete it from the contract.

Copy-paste prompt: dual-publish schema migration

Add a required `currency` field (ISO 4217) to the PaymentCompleted event, consumed by
order-service, notification-service and analytics-service.
Produce a migration plan and the first commit only:
1. contracts/payment-events.yaml: add PaymentCompleted v2 next to v1; do not remove v1.
2. A consumer adapter that accepts v1 and v2; for v1 it fills currency from the
   payment record, never from a hard-coded default.
3. Tests: a v1 fixture and a v2 fixture both produce the same domain object.
4. A counter payment_completed_consumed_total{schema_version} so we can see when v1 hits zero.
List the deploy order for the four services and the metric condition that allows step 5
(removing v1). Stop after the contract change for my review.

How do you coordinate one feature across several repositories?

A feature such as loyalty points touches customer, order, payment, and notification. Ask for the contracts and the dependency order first, in one planning session, then run one bounded implementation task per repository.

Copy-paste prompt: contract-first cross-service plan

We are adding loyalty points across customer, order, payment and notification services.
Step 1 only: write the contract changes (OpenAPI and AsyncAPI) to contracts/loyalty/,
and a plan that lists, per service: the change, the Pact contract tests it adds or
updates, and which services must deploy before it. Mark any change that is not
backward compatible. Do not implement any service.

After you approve the contracts, the tools diverge in how they fan out. In the Claude Code and Codex loops, ../contracts is a local checkout of the shared contracts repository (org/service-contracts) that holds the approved loyalty/ changes, next to the service repositories:

Claude Code

Drive each repository headless, with tool access narrowed to what the task needs:

for svc in customer order payment notification; do
  (cd "../$svc-service" && claude -p \
    "Implement the loyalty-points changes for this service per ../contracts/loyalty/. Add or update the Pact tests the plan lists. Run npm run contracts:check and npm test; stop and report if either fails." \
    --add-dir ../contracts \
    --allowedTools "Read" "Edit" "Write" "Bash(npm test:*)" "Bash(npm run contracts:check)" \
    --disallowedTools "Edit(../contracts/**)" "Write(../contracts/**)")
done

--add-dir gives the agent tool access to ../contracts, and the allowed Edit and Write would apply there too. The --disallowedTools rules block writes to the approved contracts, and deny rules take precedence over allow rules. Keep a CODEOWNERS rule or a CI check on the contracts repository as the backstop, and reject any run that touches them.

Codex

Run one non-interactive task per repository with the :workspace permission profile. There is no --add-dir here: in Codex it makes the directory writable, and the contracts only need to be read, so the approved contracts stay outside what the agent can change:

for svc in customer order payment notification; do
  codex -a never exec -C "../$svc-service" \
    -c default_permissions=':workspace' \
    "Implement the loyalty-points changes for this service per ../contracts/loyalty/. Add or update the Pact tests the plan lists. Run npm run contracts:check and npm test; stop and report if either fails."
done

To run the tasks remotely instead, use Codex cloud tasks (codex cloud, experimental in 0.157.1), one task per repository.

Cursor

Start one Cloud Agent per repository with the approved plan as its prompt, so each repository’s change is reviewed on its own. Implement in the dependency order the plan gives, and do not start a consumer until the provider’s contract tests are green. For the Cursor-specific setup, see multi-repository workflows in Cursor. For Cursor-specific microservice patterns, see microservices in Cursor.

For more than a handful of repositories, see orchestration patterns for fan-out, retries, and merge order.

Which MCP servers give the agent live telemetry?

An agent that can query errors, metrics, and cluster state reasons from evidence instead of from your screenshot. Keep these servers read-only in development work: the agent investigates, and you or CI change production.

Server	What it gives the agent	Package or endpoint (checked 2026-09-26)
Sentry (official)	Issues, events, traces, releases	Remote https://mcp.sentry.dev/mcp (OAuth); @sentry/mcp-server 0.42.0 for self-hosted Sentry
Grafana (official)	Dashboards, Prometheus and Loki queries, incidents	PyPI mcp-grafana 1.6.3 (uvx mcp-grafana), or Docker grafana/mcp-grafana; --disable-write makes it read-only
Kubernetes (community, containers org)	Pods, logs, events across kubeconfig contexts	npm kubernetes-mcp-server 0.0.67; runtime settings such as read_only = true live in a TOML file passed with --config

To keep the Kubernetes server read-only and limited to its core tools, save these two lines as ~/.config/k8s-mcp.toml:

read_only = true
toolsets = ["core"]

Claude Code

claude mcp add --transport http sentry https://mcp.sentry.dev/mcp
claude mcp add grafana -e GRAFANA_URL="$GRAFANA_URL" \
  -e GRAFANA_SERVICE_ACCOUNT_TOKEN="$GRAFANA_SERVICE_ACCOUNT_TOKEN" \
  -- uvx mcp-grafana --disable-write
claude mcp add kubernetes -- npx -y kubernetes-mcp-server@latest --config ~/.config/k8s-mcp.toml

Codex

codex mcp add sentry --url https://mcp.sentry.dev/mcp && codex mcp login sentry
codex mcp add grafana --env GRAFANA_URL="$GRAFANA_URL" \
  --env GRAFANA_SERVICE_ACCOUNT_TOKEN="$GRAFANA_SERVICE_ACCOUNT_TOKEN" \
  -- uvx mcp-grafana --disable-write
codex mcp add kubernetes -- npx -y kubernetes-mcp-server@latest --config ~/.config/k8s-mcp.toml

Cursor

Add the servers in Cursor’s MCP settings in the mcpServers JSON shape. The remote Sentry server needs only its URL; Grafana and Kubernetes take the same command and arguments as the Claude Code lines, and ${env:NAME} reads each value from your environment:

{
  "mcpServers": {
    "sentry": { "url": "https://mcp.sentry.dev/mcp" },
    "grafana": {
      "command": "uvx",
      "args": ["mcp-grafana", "--disable-write"],
      "env": {
        "GRAFANA_URL": "${env:GRAFANA_URL}",
        "GRAFANA_SERVICE_ACCOUNT_TOKEN": "${env:GRAFANA_SERVICE_ACCOUNT_TOKEN}"
      }
    },
    "kubernetes": {
      "command": "npx",
      "args": ["-y", "kubernetes-mcp-server@latest", "--config", "~/.config/k8s-mcp.toml"]
    }
  }
}

See MCP setup in Cursor for where the file lives and how to pass environment variables.

Look-alike MCP packages

On 2026-09-26, @sentry/mcp, @grafana/mcp, @kubernetes/mcp-server, and @docker/mcp-gateway did not exist on npm, and @leval/mcp-grafana and docker-mcp were community packages, not the vendors’. Run npm view <package> or check the vendor’s GitHub repository before you add a server. Read the token from your shell environment or a secret store; never paste it into the command. More servers are in observability MCP servers and infrastructure MCP servers.

Copy-paste prompt: MCP-grounded triage of a cross-service error

Using the Sentry MCP server, find the three unresolved issues in project payment-service
with the most events in the last 6 hours. For the top one, open the stack trace and the
linked trace, and name the downstream call that failed and the service that owns it.
Then use the Grafana MCP server to query that service's p95 latency and 5xx rate for the
same window. Explain the root cause with the evidence you found. Propose a fix as a diff
plus the test that would have caught it. Do not apply anything.

How do you know the change works without reading every diff?

Each risk in a distributed change has a check that fails on its own. Reading the diff is for the contract and the test files; the rest is proved by gates.

Risk	Check that proves it	Runs where	Who signs off
Breaking API or event change	contracts:check: spec lint plus Pact tests in consumer and provider pipelines	Every PR	The service owner approves the contract commit
Client out of step with the contract	Generated clients, and the contract audit table above	Every PR	CI (build fails)
Missing or double compensation	A test per saga step that injects a failure in the next step and asserts exactly-once compensation	Every PR	Author; reviewer checks that the test exists and fails without the code
Broken trace across a hop	Trace-continuity test with a known traceparent	Every PR on consumers and producers	CI
Consumers still on the old schema	Per-consumer schema_version counter	Staging and production	Tech lead gates the removal of the old version
Bad deploy	Canary with automatic rollback on error rate and latency	Deploy pipeline	CI; on-call owns the threshold

The test that proves a compensation is only as good as its assertion, so require the agent to show the failing run before the passing one. For tests you did not read, see how strong your oracle is; for rollout gates, see progressive delivery.

What breaks in agent-built distributed systems?

1. The trace dead-ends at an async boundary. A request shows two hops in Jaeger, then vanishes. The consumer never extracted the trace context from the message headers. Search the consumer for header extraction, add it with the trace-safe consumer prompt, and keep the test that asserts the same traceId on both sides. Do not accept “I added tracing” without that test.
2. A saga leaves orphaned state. The payment succeeded and the inventory reservation was never released. Reproduce by injecting a failure in the step after the one you suspect, and assert the compensation runs exactly once. Rebuild that step with the failing-test-first prompt.
3. The agent treats the service as standalone. Generated code invents an endpoint shape or skips the retry and dead-letter conventions. The repository is missing its integration section in CLAUDE.md, AGENTS.md, or the Cursor rule; add it and name the contract files before you blame the model.
4. The agent makes a test pass by weakening it. A changed expect, a new skip, or a widened timeout appears next to the fix. Protect contract and saga tests from agent edits (a CODEOWNERS rule or a hook that blocks writes to them) and see protecting the oracle.
5. A contract change breaks consumers in production. The dual-publish phase was skipped. Roll back the producer, publish both versions, and remove the old one only when its consumer counter reads zero.
6. The design is a distributed monolith. Two services write the same table, or three must deploy together. Agents do not flag this on their own. Ask: “List every place two services share a database, a write path, or a lockstep deploy,” and resolve those before you split further. Boundary design belongs in domain-driven design with AI.
7. An MCP server connects but returns nothing. The token is missing or expired, an environment variable name is wrong (GRAFANA_SERVICE_ACCOUNT_TOKEN, not the deprecated GRAFANA_API_KEY), or the Sentry OAuth login never completed. Run claude mcp list or codex mcp list, re-run the login, and if npm view shows an unfamiliar publisher, you installed a look-alike.

Where to go next with microservices

• Integration test patterns: contract tests and service-boundary checks without brittle mocks
• API development with AI: designing and evolving the single-service APIs these contracts describe
• Monitoring and observability: dashboards, alerts, and the MCP debugging loop in production
• Incident response: agent-assisted on-call when a cross-service failure pages you
• CI/CD pipelines: wiring contracts:check and canary rollback into the pipeline
• Monorepo workflows: the alternative when you can keep services in one repository