Monitoring & Logging Patterns

Your API is throwing intermittent 500s. The Grafana dashboards are all green, the error logs don’t line up with the spans, and your PM wants an ETA. The painful truth: observability config is fiddly, easy to copy wrong, and the one trace you need got sampled away an hour ago.

This recipe shows how to use Cursor, Claude Code, and Codex to generate the instrumentation and config that catches these failures — and how to connect the AI directly to Grafana and Sentry so it can read your live telemetry while it debugs.

What You’ll Walk Away With

• A copy-paste prompt that auto-instruments an Express or FastAPI service with OpenTelemetry and exports OTLP to your collector
• Prompts that generate Prometheus recording rules, SLO burn-rate alerts, and an AlertManager routing tree you can review line by line
• A Logstash pipeline prompt that parses JSON logs, redacts PII, and correlates trace_id so logs link back to traces
• The Grafana and Sentry MCP setup that lets the agent query your real dashboards and issues instead of guessing
• The failure modes that bite in production — cardinality explosions, dropped spans, sampling that hides the trace you need

Wire Up the Observability MCP Servers First

Before generating config, give the agent eyes on your live telemetry. With the Grafana and Sentry MCP servers connected, the model can read dashboards, query datasources, and pull stack traces — so its suggestions are grounded in your actual data, not generic boilerplate.

Cursor

Add to .cursor/mcp.json. Sentry’s official server is remote (hosted); Grafana ships an official Go binary you run locally and point at your instance:

{
  "mcpServers": {
    "sentry": { "url": "https://mcp.sentry.dev/mcp" },
    "grafana": {
      "command": "mcp-grafana",
      "env": {
        "GRAFANA_URL": "https://grafana.example.com",
        "GRAFANA_API_KEY": "${GRAFANA_SERVICE_ACCOUNT_TOKEN}"
      }
    }
  }
}

Claude Code

Sentry is the official remote server (authenticate with /mcp after adding). Grafana’s official grafana/mcp-grafana binary runs over stdio:

# Official remote Sentry MCP — auth via /mcp
claude mcp add --transport http sentry https://mcp.sentry.dev/mcp

# Official Grafana Labs MCP (the mcp-grafana binary / mcp/grafana Docker image)
claude mcp add grafana \
  --env GRAFANA_URL=https://grafana.example.com \
  --env GRAFANA_API_KEY=$GRAFANA_SERVICE_ACCOUNT_TOKEN \
  -- mcp-grafana

Codex

Codex stores MCP entries in ~/.codex/config.toml. Use --url for the hosted Sentry server and a stdio command for Grafana:

# Hosted Sentry MCP (OAuth: codex mcp login sentry)
codex mcp add sentry --url https://mcp.sentry.dev/mcp

# Grafana Labs stdio server
codex mcp add grafana \
  --env GRAFANA_URL=https://grafana.example.com \
  --env GRAFANA_API_KEY=$GRAFANA_SERVICE_ACCOUNT_TOKEN \
  -- mcp-grafana

Note

Use the official servers, not lookalikes. Sentry’s is the remote https://mcp.sentry.dev/mcp (or the @sentry/mcp-server npm package for self-hosting) — not the abandoned sentry-mcp. Grafana’s is grafana/mcp-grafana from Grafana Labs, not third-party npm reimplementations. For Elasticsearch, Elastic publishes @elastic/mcp-server-elasticsearch. Squatted or hobby MCP packages go stale fast and can leak credentials.

The payoff: instead of pasting a stack trace into chat, you ask the agent to fetch it.

Tip

Debug a live incident with the Sentry MCP connected:

Using the Sentry MCP, find the top 3 unresolved issues in project api-prod from the last 6 hours by event count. For the highest-frequency one, pull the latest event’s stack trace and breadcrumbs, identify which release introduced it (compare first_seen to our deploy times), and propose the smallest code fix. Show me the suspect function before you suggest changes.

Instrument the Service (Where Most Bugs Actually Start)

Dashboards are downstream of instrumentation. If a service emits the wrong attributes or no trace context, no amount of Grafana panels will save you. This is the highest-leverage thing to get right, and it’s identical regardless of which agent you use — the prompt is what matters.

Tip

Auto-instrument a Node/Express service with OpenTelemetry:

Instrument this Express service with OpenTelemetry using the OTLP/HTTP exporter (@opentelemetry/exporter-trace-otlp-proto), not the deprecated Jaeger exporter. Auto-instrument http, express, pg, and ioredis; disable fs instrumentation. Set service.name, service.version, and deployment.environment from env vars. Redact Authorization and Cookie headers on spans. Export to http://otel-collector:4318/v1/traces with a BatchSpanProcessor. Add a SIGTERM handler that flushes spans before exit.

What the agent generates is the standard SDK bootstrap — the part worth reviewing is the exporter and shutdown:

// tracing.js — load with: node --require ./tracing.js server.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-proto');
const { resourceFromAttributes } = require('@opentelemetry/resources');
const {
  ATTR_SERVICE_NAME,
  ATTR_SERVICE_VERSION,
} = require('@opentelemetry/semantic-conventions');

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: process.env.SERVICE_NAME ?? 'api-service',
    [ATTR_SERVICE_VERSION]: process.env.SERVICE_VERSION ?? '0.0.0',
    'deployment.environment': process.env.ENVIRONMENT ?? 'production',
  }),
  // Jaeger ingests OTLP natively since v1.35 — no Jaeger exporter needed.
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTLP_ENDPOINT ?? 'http://otel-collector:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations({
    '@opentelemetry/instrumentation-fs': { enabled: false },
  })],
});

sdk.start();
process.on('SIGTERM', () => sdk.shutdown().finally(() => process.exit(0)));

Caution

If you ask an older model for “OpenTelemetry tracing”, it will often reach for @opentelemetry/exporter-jaeger and a 14268/api/traces Thrift endpoint. OpenTelemetry dropped Jaeger exporter support in July 2023 and the Jaeger client libraries are archived. Always export OTLP (port 4318 for HTTP, 4317 for gRPC). Jaeger has accepted native OTLP since v1.35 — point your exporter at the collector, not at a Jaeger Thrift port.

Generate Prometheus Rules and SLO Alerts

Prometheus rule files are where AI assistance shines — the syntax is finicky and the PromQL is easy to get subtly wrong. Keep the prompt opinionated so you get RED-method recording rules plus burn-rate alerts, not a wall of every metric imaginable.

Tip

Generate SLO-based alerting rules:

Write a Prometheus rules file for an HTTP API. Add recording rules for request rate, 5xx error ratio, and p95/p99 latency from http_request_duration_seconds_bucket, all grouped by job. Then add multi-window burn-rate alerts for a 99.5% availability SLO: a fast-burn alert (1h and 5m windows, 14.4x budget burn, severity critical) and a slow-burn alert (6h and 30m windows, 6x burn, severity warning). Include summary, description, and a runbook_url annotation on each.

The generated recording rules look like this — short, reviewable, and the foundation every alert builds on:

rules/api.yml

groups:
  - name: api_recording
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_errors:ratio5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          / sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_latency:p95_5m
        expr: |
          histogram_quantile(0.95,
            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

Caution

Watch the cardinality. Models love to add high-cardinality labels — user_id, request_id, full URL paths with IDs baked in. Each unique label combination is a separate time series; a label like user_id can multiply your series count by millions and OOM Prometheus. When you review generated rules, reject any label that isn’t bounded to a small, known set (status code, method, route template, job). Ask the agent explicitly: “audit these metrics for unbounded label cardinality.”

For AlertManager routing, hand the agent your escalation policy in plain English and let it produce the tree:

Tip

Generate an AlertManager routing config:

Write an alertmanager.yml. Route severity: critical to PagerDuty with group_wait: 0s and repeat_interval: 1h. Route alerts where service matches ^(postgres|redis|mongodb) to a #db-alerts Slack channel. Mute severity: warning alerts on weekends and outside 09:00–18:00 on weekdays via mute_time_intervals. Add an inhibit rule so a firing critical suppresses matching warning alerts for the same alertname and cluster. Reference secrets via env vars, never inline.

Parse and Correlate Logs

The single highest-value logging move is making logs link to traces. If every log line carries trace_id, you click from a Grafana span straight to the surrounding logs. The prompt below produces a Logstash pipeline that does exactly that.

Tip

Generate a Logstash pipeline with trace correlation and PII redaction:

Write a logstash.conf with a Beats input on 5044 (TLS) and a Kafka input on the application-logs topic. Parse JSON-formatted log bodies, promote trace_id and span_id to top-level fields so logs correlate with traces, and convert status to an integer. Redact credit-card and email patterns in the message with gsub. Drop DEBUG logs when environment == "production". Output to Elasticsearch using ILM with rollover alias logs, and tee anything where log_level == "ERROR" to a dead_letter file.

The trace-correlation block is the part to verify — make sure the field names match what your OpenTelemetry SDK emits:

filter {
  if [message] =~ /^\{/ {
    json { source => "message" target => "parsed" }
    mutate {
      rename => {
        "[parsed][trace_id]" => "trace_id"
        "[parsed][span_id]"  => "span_id"
        "[parsed][level]"    => "log_level"
      }
    }
  }
  if [environment] == "production" and [log_level] == "DEBUG" { drop {} }
}

Note

Prefer Loki over the ELK stack? The prompts transfer directly — swap “Logstash pipeline” for “Promtail/Alloy config” and “Elasticsearch output” for “Loki push endpoint”. The trace-correlation logic (trace_id as a label/field) is the same idea, and Loki’s LogQL pairs naturally with the Prometheus rules above since both live in Grafana.

Tail-Sampling in the Collector

The default head sampling drops traces blindly — which is how the one trace you needed disappears. Tail sampling decides after a trace finishes, so you can keep 100% of errors and slow requests while sampling the boring stuff at 5%.

Tip

Configure tail sampling in the OpenTelemetry Collector:

Add a tail_sampling processor to my OTel Collector config with three policies combined: keep 100% of traces containing any span with status_code = ERROR, keep 100% of traces with total latency over 1000ms, and sample everything else at 5% probabilistic. Set decision_wait: 30s. Also add an attributes processor that deletes http.request.header.authorization and db.statement before export. Wire both into the traces pipeline exporting OTLP to our backend on 4317.

Caution

Tail sampling needs every span of a trace to reach the same collector instance. If you run multiple collector replicas behind a round-robin load balancer, spans from one trace scatter across replicas and the sampler makes decisions on partial traces — silently dropping data. Use a loadbalancing exporter (trace-ID-aware routing) in a two-tier collector setup, or run a single tail-sampling collector. Ask the agent to “explain how this scales to N replicas” before you deploy it.

When This Breaks

1. Spans never show up. Check the exporter endpoint and protocol mismatch first — OTLP/HTTP is 4318, OTLP/gRPC is 4317, and the proto vs http/json exporter must match the receiver. Run the collector’s debug exporter (verbosity: detailed) to confirm spans arrive before blaming the backend.

2. Prometheus is OOMing or scrapes are slow. You have a cardinality explosion. Query topk(20, count by (__name__)({__name__=~".+"})) and prometheus_tsdb_head_series to find the offender, then drop the culprit label with a metric_relabel_configs rule. This is almost always a label the AI added that you didn’t catch in review.

3. Alerts fire late or in storms. Re-check for:, group_wait, and repeat_interval. A burn-rate alert with too long a window won’t page until you’ve already blown the budget; group_wait: 0s on everything turns a cluster outage into hundreds of pages.

4. Logs don’t correlate to traces. The trace_id field name from your SDK doesn’t match what the pipeline expects, or it’s nested. Log one raw event and confirm the exact path before trusting the rename/mutate block.

5. The trace you needed got sampled away. Head sampling dropped it. Move error/latency decisions to tail sampling in the collector (above), and keep errors at 100%.

What’s Next

SQL Patterns Recipes Connect a database MCP server and let the agent profile slow queries against real data.

CI/CD Pipeline Recipes Wire these alerts and SLOs into deploy gates and automated rollbacks.

Debugging Patterns The systematic prompt patterns for turning telemetry into a root cause.

MCP Best Practices How MCP works across Cursor, Claude Code, and Codex, and how to vet a server.