Distributed Systems Development with AI

Distributed systems development with AI works contract-first: service interfaces are defined before implementation, so Cursor, Claude Code, and Codex generate and validate code against a shared specification instead of guessing at another service’s behavior. Saga steps, trace propagation, and dual-publish schema migrations are then built and verified one at a time, because the failure modes live between services.

You change one field on the Order service’s API and three other services start returning 500s in staging. The trace is incomplete because two services never propagated the trace context, the saga that processes payments silently skipped its compensation step, and your on-call dashboard shows green while customers can’t check out. Three services, three repositories, three different teams — and the AI tool you are driving can only see the repo you have open.

That gap is where AI assistants are most useful and most dangerous: they generate plausible cross-service scaffolding fast, but “generate the entire production system” gets you code you can’t verify. This guide covers the parts AI is genuinely good at — drafting service skeletons, propagating trace context, writing the boring compensation logic — while keeping the verification loop tight enough that you’d ship the result.

What this distributed-systems workflow gives you

Per-tool rules files that give a single-repo agent the cross-service integration context it is otherwise missing
Copy-paste prompts for designing service boundaries, extending an existing contract endpoint-first, and auditing a client against the contract it claims to implement
A saga recipe that builds one step at a time, each with its compensation and a failing test
A dual-publish schema-migration plan that moves producers and consumers through a version change without breaking each other
Trace-driven debugging and incremental OpenTelemetry instrumentation you can verify in Jaeger before you template it
The real, verified MCP servers for monitoring (Sentry, Grafana, Dynatrace) and infrastructure (Docker, Kubernetes, AWS) — with the exact install commands
Recovery steps for the failure modes that actually bite: broken trace context, missing saga compensation, contract drift, and MCP auth failures

Why one open repo is never enough context

Microservices split a system across repositories, languages, and teams. AI tools see one repository at a time, which means the model’s default assumption — that this service is a standalone application — is wrong in exactly the places that break production. The fix is to encode the cross-service contracts and conventions inside each repository, so the integration context is always loaded whether or not you remember to mention it.

Store contract definitions in the service repository and name them in a rules file:

This service (order-service) communicates with:
- payment-service: REST API, OpenAPI spec at /contracts/payment-api.yaml
- inventory-service: Events via RabbitMQ, schemas at /contracts/inventory-events.json
- notification-service: Events via RabbitMQ, schemas at /contracts/notification-events.json

When implementing any integration:
1. Always read the relevant contract file first
2. Generate client code from the contract, do not hand-write it
3. Include retry logic with exponential backoff for all HTTP calls
4. Include dead-letter queue handling for all event consumers

Use @contracts/payment-api.yaml to bring a specific contract into the conversation.

Claude Code reads contract files directly, so point CLAUDE.md at the directory and state the regeneration rule:

Microservice: order-service
Contracts directory: /contracts/
- payment-api.yaml (OpenAPI 3.1) - payment-service REST API
- inventory-events.json (AsyncAPI 2.6) - inventory-service event schemas
- notification-events.json (AsyncAPI 2.6) - notification-service events

Integration rules:
- All HTTP clients must use the generated SDK in /src/clients/
- Regenerate clients when contracts change: npm run generate-clients
- All event publishers must validate against the schema before publishing
- Circuit breaker pattern required for all external service calls

Codex can reach multiple repositories through its GitHub integration, so name the siblings and where the shared contracts live:

This is part of a microservices architecture. Related repos:
- org/payment-service - Payment processing
- org/inventory-service - Stock management
- org/notification-service - User notifications

Shared contracts are in org/service-contracts repo.
When making changes that affect service boundaries:
1. Check the contract in org/service-contracts first
2. Update the contract if needed (creates PR to service-contracts)
3. Implement the change in this service
4. Note any downstream services that need updates

MCP servers that actually exist

The other half of the context problem is live infrastructure: letting the AI query it beats letting it guess. But the ecosystem is full of look-alike npm packages — sentry-mcp is a low-traffic stub, not Sentry’s server. Use these verified servers. MCP setup is identical across Cursor, Claude Code, and Codex: all three read the same server definitions (.mcp.json for Claude Code, .cursor/mcp.json for Cursor, ~/.codex/config.toml for Codex), so the commands below apply to whichever tool you drive.

Monitoring and observability

Sentry (errors, traces, releases) — use the official hosted server with OAuth, no token to manage:
Terminal window
```
claude mcp add --transport http sentry https://mcp.sentry.dev/mcp
```
For a self-hosted Sentry, the official npm package is @sentry/mcp-server:
Terminal window
```
claude mcp add sentry -- npx -y @sentry/mcp-server@latest --access-token=YOUR_TOKEN
```
Grafana (dashboards, Loki/Prometheus queries, incidents) — the official server is grafana/mcp-grafana, a Go binary distributed via Docker (there is no mcp-grafana npm package):
Terminal window
```
claude mcp add grafana -- docker run --rm -i \
  -e GRAFANA_URL=http://localhost:3000 \
  -e GRAFANA_SERVICE_ACCOUNT_TOKEN=YOUR_TOKEN \
  grafana/mcp-grafana -t stdio
```

Dynatrace (APM, AI anomaly detection) — the official package is published by the Dynatrace OSS org and needs Node 22.10+:

DT_ENVIRONMENT=https://YOUR.apps.dynatrace.com \
  claude mcp add dynatrace -- npx -y @dynatrace-oss/dynatrace-mcp-server@latest

Containers and infrastructure

Docker — the official MCP ships with Docker Desktop’s MCP Toolkit; you run the gateway rather than an npm package:
Terminal window
```
claude mcp add docker -- docker mcp gateway run
```
In Cursor, add a command-type server in Settings → MCP pointing at the same docker mcp gateway run.
Kubernetes — kubernetes-mcp-server is a real package; it uses your current kubeconfig context:
Terminal window
```
claude mcp add k8s -- npx -y kubernetes-mcp-server@latest
```
AWS — AWS Labs publishes purpose-specific servers (not one monolithic image). Pick the one you need and rely on the standard AWS credential chain rather than inlining keys:
Terminal window
```
claude mcp add aws-api -- uvx awslabs.aws-api-mcp-server@latest
```
Browse the full catalog at awslabs.github.io/mcp. For Google Cloud, deploy a custom MCP server on Cloud Run—see cloud.google.com/run/docs.

Designing service boundaries

AI drafts bounded-context proposals quickly, but the boundaries are a business decision — treat the output as a first draft to argue with, not a verdict. Start narrow: ask for boundaries plus the reasoning, so you can spot where the model conflated a technical layer with a domain.

I have an e-commerce monolith with these features: user management,
product catalog, inventory tracking, order processing, payments,
shipping, notifications. Propose microservice boundaries using
domain-driven design. For each proposed service, give me:
(1) the bounded context and the single team that would own it,
(2) the data it owns exclusively,
(3) which other services it talks to and whether that's sync or async.

Flag any place where two contexts share a database or a write path—those
are coupling risks I need to resolve before splitting.

When you’ve agreed on boundaries, design one service at a time. Resist “generate all services” — you can’t review a dump of seven services, and the contracts between them are where bugs hide.

For the Order Processing context, design the service contract only (not
the implementation yet): the REST endpoints with request/response
schemas, the domain events it publishes (OrderCreated, OrderConfirmed,
OrderCancelled) with their payloads, and the events it consumes from
Payment and Inventory. Output the OpenAPI spec and the event schemas as
separate files. Do not generate business logic—I want to review the
contract first.

Changing an existing contract, endpoint first

The prompt above designs a contract from nothing. Day to day you are extending one that exists, and the discipline is the same in reverse: change the spec, let the spec generate the code, and review the spec before a line of implementation appears.

I need to add a refund endpoint to the payment service. Work contract-first:
1. Read the existing OpenAPI spec at /contracts/payment-api.yaml
2. Add a POST /payments/{paymentId}/refund endpoint to the spec with:
   - Request body: { amount: number, reason: string, idempotencyKey: string }
   - Response 200: { refundId: string, status: "pending" | "completed", amount: number }
   - Response 404: Payment not found
   - Response 409: Refund already exists for this idempotency key
   - Response 422: Amount exceeds original payment
3. Generate the Express route handler from the contract
4. Generate the client SDK method from the contract
5. Generate integration test stubs from the contract

Show me the updated contract before generating any implementation code.

Contracts drift the moment someone hand-edits a client. AI is good at catching that, and it only needs your side of the boundary to do it — the contract file is the other side. Each tool has a natural shape for the audit:

Compare our order-service HTTP client for the payment service
against the payment-api.yaml contract:
1. Are we handling all documented error codes?
2. Are we sending all required headers?
3. Are we respecting rate limits and timeouts from the spec?
4. Are there any fields we're ignoring in responses that we should handle?

claude "Read /contracts/payment-api.yaml and /src/clients/payment-client.ts.
Perform a contract compliance audit:
- List every endpoint in the contract and whether our client implements it
- Check error handling for every documented error response
- Verify request/response types match the schema
- Check timeout and retry configurations against SLA requirements
Output as a compliance checklist with pass/fail for each item."

Audit contract compliance for the order-service against all its upstream contracts.
For each contract in /contracts/:
1. Find the corresponding client implementation in /src/clients/
2. Verify every endpoint, error code, and schema field is handled
3. Check for missing retry logic, circuit breakers, and timeout handling
4. Create issues for any violations found

Run the audit in CI, not just on demand: npm run validate-contracts failing the build is what stops drift from reaching production.

Saga patterns you can actually verify

The saga pattern is where AI-generated distributed code most often looks right and is wrong. The failure mode is always the same: the happy path is fine, but a compensation step is non-idempotent, or a timeout budget is missing. The fix is to build one step at a time, each with its compensation and a failing test first, then watch it go green.

Open the Order service repo and switch to Agent mode. Ask for one saga step plus a failing test, run the test in Cursor’s terminal, and only accept the diff once it’s green. Use a checkpoint before each step so you can roll back a bad compensation without losing the prior steps. Cursor’s inline diff view makes it easy to spot when the model “fixed” the test by weakening the assertion instead of the code.

Drive it from the terminal so the test run is part of the loop. Claude Code can run the test, read the failure, and iterate without you copy-pasting output:

claude "Implement step 3 of the order saga (process payment) plus its
compensation (refund). Write a failing test that asserts the refund fires
when step 4 throws, then make it pass. Run the test with `npm test -- saga`
and show me the diff before committing."

Add a hook in .claude/settings.json that runs the saga test suite on every edit to saga/, so a regression in an earlier step surfaces immediately.

Use a dedicated git worktree so the saga work is isolated in its own local checkout, then have Codex run the suite per step. ChatGPT desktop can create an optional managed worktree; CLI and IDE tasks use a checkout you select:

codex --sandbox workspace-write -c approval_policy=on-request "Implement the payment step of the order
saga with an idempotent compensation. Add a test that injects a failure at
the inventory step and asserts payment is refunded exactly once on retry.
Run the suite and stop for my review before applying."

Implement one step of the orchestrator-based order saga: "reserve
inventory." Requirements: (1) the step is idempotent—calling it twice
with the same order ID reserves once; (2) it has a compensation that
releases the reservation; (3) a 30s timeout budget, after which the
orchestrator triggers compensation.

Write the failing test first: simulate the payment step (the next step)
throwing, and assert the inventory reservation is released exactly once.
Then implement until the test passes. Show me the test and the diff—do
not add the remaining saga steps yet.

Tie every step to an observable check: after the model claims a step works, run the one test that proves the compensation fires. If you can’t articulate the test, you can’t trust the code.

Inter-service communication and trace context

Service mesh and gateway configs are high-leverage for AI—but again, incrementally. Start with the smallest config that you can verify with a single command (curl, istioctl analyze), then layer on canary weights and circuit breakers.

Generate the minimal Istio config to enforce mTLS between the order and
payment services: one PeerAuthentication in STRICT mode plus the
DestinationRule. Give me the exact `istioctl analyze` and `curl` commands
to confirm mTLS is active and that plaintext calls are rejected. Once
that's verified, I'll ask you to add a 90/10 canary VirtualService—don't
add traffic splitting or circuit breakers yet.

For event-driven flows, the recurring production bug is a broken trace: a service consumes a Kafka message but never extracts and re-injects the trace context, so the trace dead-ends. When you ask AI to wire up consumers, make context propagation an explicit, tested requirement—not an afterthought.

Generate a Node.js Kafka consumer template for the Order domain using
KafkaJS and OpenTelemetry. The critical requirement: extract the trace
context from incoming message headers and make it the active context for
the handler span, then inject the current context into headers of any
message it produces. Include a test that publishes a message with a known
traceId and asserts the downstream produced message carries the same
traceId. Handle poison messages with a dead-letter topic.

When something does break across a boundary, work from the trace backward rather than from the service you happen to have open. Hand the AI the trace timeline and the one consumer you suspect, and make it reason about why it is intermittent — that question is what separates a real diagnosis from a plausible one.

I have a distributed trace showing this sequence:
1. web-gateway receives POST /checkout (200ms)
2. order-service creates order (150ms)
3. payment-service charges card (800ms, success)
4. inventory-service.reserve event published (50ms)
5. inventory-service.reserve event consumed (TIMEOUT after 30s)
6. order-service times out waiting for inventory confirmation

The inventory service logs show the event was received but processing hung.
Analyze our inventory service event consumer at /src/consumers/reserve.consumer.ts
and identify:
- Where the processing could hang (database locks, external calls, deadlocks)
- Why it only fails intermittently (race conditions, connection pool exhaustion)
- What observability we're missing (metrics, structured logs, health checks)

Distributed schema migrations

Changing a data format that crosses a service boundary is not a code change, it is a choreography. Producers and consumers deploy independently, so the only safe path is one where both old and new formats are valid at the same time.

Define the new schema version

Add the new schema alongside the old one. Do not replace it yet.
Update producers to publish both versions

The producing service sends events in both old and new formats for the transition period.
Update consumers to accept both versions

Each consuming service handles both schema versions gracefully.
Verify all consumers are updated

Monitor that no service is still consuming the old format — a metric, not an assumption.
Remove the old schema

Only after every consumer has migrated, stop producing the old format and delete it.

I need to add a `currency` field to the PaymentCompleted event that's consumed by
order-service, notification-service, and analytics-service.

Generate a backward-compatible migration plan:
1. Update the event schema in /contracts/payment-events.json to v2 with currency field
2. Modify the payment-service publisher to emit both v1 and v2 events
3. Generate a v2 event consumer that handles both versions (check for currency field, default to "USD" if missing)
4. Create a test that verifies backward compatibility with v1 events
5. Add a metric to track which consumers are still on v1

Start with the contract change and show me before proceeding.

Observability: instrument incrementally

Modern observability has moved beyond dashboards to AI-driven anomaly detection and topology-aware root-cause analysis—but you still earn it one service at a time. The “instrument 8 services and 3 databases in one prompt” approach produces config you can’t validate. Instrument one service end to end, confirm a span shows up in Jaeger, then template it.

Add OpenTelemetry auto-instrumentation to the Order service
(Node.js/Express). Auto-instrument the HTTP server and the Postgres
client, add one custom span around the order-creation business logic, and
export traces to a local Jaeger via OTLP. Give me the exact steps to send
one request and confirm the trace appears in Jaeger at localhost:16686,
including the custom span. Once I confirm, I'll ask you to extract this
into a shared instrumentation module the other services import.

Once that first trace lands, the wider instrumentation is a template rather than a gamble, and it is worth asking for the whole shape at once — spans, correlated logs, health checks, and metrics — because each piece is now verifiable against a working baseline.

Set up distributed tracing for our Express-based microservice:
1. Add OpenTelemetry instrumentation with automatic trace propagation
2. Include custom spans for: database queries, external HTTP calls, message queue operations
3. Add structured logging with trace ID correlation (use pino)
4. Create a health check endpoint that verifies all downstream dependencies
5. Add Prometheus metrics for: request duration, error rate, active connections, queue depth

Follow our existing patterns in /src/middleware/ for consistency.
Include the Docker Compose configuration for a local Jaeger instance.

With the Grafana and Sentry MCP servers connected, you can close the loop without leaving your editor: ask the AI to pull the actual error rate or the slowest trace for a service and reason about it, instead of you screenshotting a dashboard.

Using the Sentry MCP, find the top 3 unresolved issues for the
payment-service project in the last 6 hours, ranked by event count. For
the most frequent one, pull the stack trace and the linked trace, identify
which downstream call failed, and propose a fix as a diff. Don't apply
it—explain the root cause first so I can confirm.

Coordinating changes across repos

A feature like “loyalty points” touches Customer, Order, Payment, and Notification. The coordination problem—not the per-service code—is what makes this hard, and the three tools take genuinely different approaches.

Open all four service repos in a single multi-root workspace so the agent can see every contract at once. Design the OpenAPI/event contracts first, then use a background agent to implement each service in dependency order while you review diffs per repo. Cursor’s per-file checkpoints let you revert one service’s changes without unwinding the others. Best when you want to watch and steer each service’s diff visually.

Script the coordination. Claude Code runs headless, so you can drive each repo non-interactively and gate on contract tests:

for svc in customer order payment notification; do
  (cd "../$svc" && claude -p "Implement the loyalty-points changes per
    ../contracts/loyalty.openapi.yaml. Add Pact contract tests against the
    services you call. Stop if any contract test fails." \
    --allowedTools Read Edit Bash)
done

Sub-agents and the -p headless mode make Claude Code the strongest fit when the change is mechanical across many repos and you want it auditable in CI.

We're adding Customer Loyalty Points across Customer, Order, Payment, and
Notification services. Step 1 only: design the API contracts and event
schemas (OpenAPI + event payloads) and a dependency-ordered implementation
plan that says which service to change first and why. Add the contract
tests (Pact) that must pass between each pair before we deploy. Do not
implement any service yet—I want to review the contracts and the sequence.

When AI-assisted distributed systems break

Distributed systems fail in ways a single-service mindset misses. Here are the failure modes that actually surface with AI-assisted work and how to recover.

Trace context dead-ends at an async boundary. A request shows up in Jaeger for two hops then vanishes. The consumer didn’t extract the trace context from message headers. Search the consumer for context extraction; if it’s missing, ask the AI to add header-based propagation and a test that asserts a known traceId survives the hop (see the Kafka prompt above). Don’t trust “I added tracing”—verify the traceId end to end.
A saga leaves orphaned state. Payment succeeded, inventory was never released after a downstream failure. The compensation is missing or non-idempotent. Reproduce by injecting a failure at the step after the one you suspect, and assert compensation fires exactly once. Rebuild that step with the failing-test-first prompt; never accept compensation logic without a test that triggers it.
The AI treats each service as a standalone app. Generated code ignores the contract, invents an endpoint shape, or skips the retry and dead-letter conventions every other service follows. The repo is missing its .cursor/rules / CLAUDE.md / AGENTS.md integration section — add it, naming the contract files explicitly, before blaming the model.
Generated client code doesn’t match the contract. Regenerate clients from contracts; do not hand-write them. Then put the compliance audit in CI so drift fails the build rather than surfacing as a 422 in production.
A contract change breaks downstream services in production. You skipped the dual-publish phase. Run both schema versions simultaneously during the migration, and use the version-tracking metric to confirm every consumer has moved before you remove the old format.
MCP server auth fails or returns nothing. The tool connects but every query errors or returns empty. Usually a missing/expired token or wrong env var (GRAFANA_SERVICE_ACCOUNT_TOKEN, DT_ENVIRONMENT, Sentry OAuth not completed). Run claude mcp list to confirm the server is connected, re-check the env vars against the install commands above, and for the hosted Sentry server re-run the OAuth flow. If npm view <pkg> shows a suspiciously low download count, you installed a look-alike—reinstall the official scoped package.
The AI generated a “distributed monolith.” Services that must deploy together, or two services writing the same table. This is a design failure the model won’t flag on its own. Ask it to audit: “List every place two services share a database, a write path, or must deploy in lockstep.” Resolve those before splitting further—shared write paths defeat the point of microservices.
Distributed debugging still takes forever. AI helps far less when it has only raw log files to correlate by hand. Invest in the observability layer first: structured logs carrying the trace ID, a real trace backend, and an MCP server that can query both. The trace-driven prompt above is only as good as the trace you can hand it.
Canary auto-rollback never triggers. The deploy went bad but stayed at 100%. The rollback threshold references a metric that isn’t being emitted, or the metric name is wrong. Confirm the golden-signal metrics exist in Prometheus/Grafana (use the Grafana MCP to query them) before relying on automated rollback, and test the rollback path in staging with a deliberately failing build.

Where to go next with microservices

Monitoring and Observability — go deeper on OpenTelemetry, Grafana dashboards, and the Sentry MCP debugging loop
Incident Response — AI-assisted on-call workflows for distributed system failures
Pipeline Automation with AI — change-aware builds and safe progressive deploys for the services you just split
Infrastructure as Code with AI Assistants — Terraform, Pulumi, and the provider MCP servers that ground AI in real state
Integration Test Patterns — contract testing and service-boundary verification without brittle mocks
API Testing — contract testing and API automation across service boundaries
Must-Have MCP Servers for Every Developer — the foundational servers behind every workflow above