Observability setup with Claude Code

Observability setup with Claude Code means having the agent add OpenTelemetry tracing, trace-correlated structured logs, Prometheus alert rules and Grafana dashboards to an existing service, while hooks, a deny rule, a verifier subagent and the Grafana MCP server check every generated file. Nobody reads the YAML line by line; promtool tests and live queries prove it.

It is 2 AM and the pager fires: “high error rate on API”. The dashboard shows a spike in 5xx responses, but no trace IDs, no log correlation and no hint of which downstream call is failing. You grep unstructured logs for 40 minutes to find a payment-provider timeout that takes two minutes to fix. This page is for the developer who owns that service and the tech lead who signs off on its alerting.

It covers the Claude Code harness around the work: plan mode, CLAUDE.md, hooks, permission rules, subagents and headless runs. The stack itself (collector, Loki, Tempo, metric naming and CI checks) is covered tool-neutrally in monitoring and observability with AI agents. To observe Claude Code’s own sessions, cost and tool calls, see agent observability.

What you’ll walk away with from Claude Code observability setup

• A plan-mode prompt that inventories the service before any instrumentation is written.
• A CLAUDE.md block that pins the observability conventions every later session follows.
• A PostToolUse hook that runs promtool and amtool after every edit, and a deny rule that keeps the alert tests out of the agent’s reach.
• Copy-paste prompts for OpenTelemetry with correlated logs, burn-rate alerts with unit tests, and a dashboard check through the Grafana MCP server.
• A verifier subagent and a headless run that produce the evidence the pull request carries.

In what order should Claude Code build your observability?

Each layer is only as good as the one beneath it. An alert on a metric name the service never emits stays silent forever, and nothing reports that it is silent. Build in this order and do not let Claude skip ahead.

Step	What Claude Code does	Claude Code mechanism	What proves it without reading it
1. Map	Inventories entry points, clients, queues and logging	Plan mode (read-only)	The inventory table you approve
2. Instrument	Adds the OpenTelemetry SDK and a structured logger	CLAUDE.md conventions	One request’s trace ID in the collector and in a log line
3. Alert	Writes rules plus promtool tests	PostToolUse hook, deny rule on the tests	Green promtool test rules with must-fire and must-stay-silent cases
4. Dashboard	Writes dashboard JSON	Grafana MCP server, read-only	A table in which every panel returns series
5. Evidence	Re-runs all checks and reports	Verifier subagent, claude -p	A JSON report attached to the pull request

Map the service before Claude writes any instrumentation

Start in plan mode, so Claude reads the code and proposes a plan without editing anything. The inventory is what you review, and it is 20 lines instead of a 600-line diff.

# Terminal, from the repository root
claude --permission-mode plan

Copy-paste prompt: observability inventory in plan mode

Inventory this service for observability. Do not edit anything.

Produce one table with these rows, each with file paths:
1. Process entry points (HTTP server, workers, cron jobs, CLI commands).
2. Inbound frameworks and their versions from the lockfile.
3. Outbound calls: HTTP clients, database drivers, caches, queues, third-party SDKs.
4. Current logging: library, call sites (count them), and any place that logs request
   bodies, headers, tokens or personal data.
5. Existing metrics, health endpoints and tracing, if any.
6. Code paths that auto-instrumentation will miss: custom HTTP clients, queue consumers,
   background jobs, anything that spawns work outside a request.

Then propose the instrumentation plan in the order: tracing, logs, metrics, alerts,
dashboards. List the packages you would install and why each one is needed.

Row 6 is the one that matters most. Auto-instrumentation covers standard HTTP servers and database drivers, and misses queue consumers and hand-rolled clients. That gap is where traces stop at a boundary later.

Pin the observability conventions in CLAUDE.md

Every later session, including the one that edits an alert six months from now, should follow the same rules. Put them in the project CLAUDE.md rather than in each prompt.

## Observability conventions

- Tracing: OpenTelemetry via src/instrumentation.ts, loaded with node --require before the app.
  Build resources with resourceFromAttributes(); the Resource class is gone in resources 2.x.
- Logging: pino only. JSON in production. Every entry carries trace_id and span_id from the
  active span. Redact password, token, secret, authorization, cookie and card fields.
- Metric names: read them from the running Prometheus before writing any query
  (curl -s localhost:9090/api/v1/label/__name__/values). Never write a metric name from memory.
- Alerts live in observability/alerts.yml. observability/alerts_test.yml is the oracle:
  never edit it. If a test fails, fix the rule or explain why the test is wrong.
- Every alert has severity, summary and runbook_url labels or annotations.
- Dashboards live in observability/dashboards/*.json and use a $service template variable.

Instrument traces and correlated logs

With the plan approved, switch to normal mode and run the bootstrap prompt. It pins the current OpenTelemetry API because models trained on older tutorials reach for the removed Resource class. Package versions current on 2026-09-26 (read with npm view): @opentelemetry/sdk-node 0.222.0, @opentelemetry/auto-instrumentations-node 0.80.0, @opentelemetry/resources 2.11.0 and pino 10.3.1.

Copy-paste prompt: OpenTelemetry plus trace-correlated pino logs

Follow the observability conventions in CLAUDE.md and the plan we agreed.

1. Create src/instrumentation.ts with @opentelemetry/sdk-node and getNodeAutoInstrumentations,
   fs instrumentation disabled, resource built with resourceFromAttributes() and
   ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION, ATTR_DEPLOYMENT_ENVIRONMENT_NAME. Export traces
   and metrics over OTLP to OTEL_EXPORTER_OTLP_ENDPOINT. Load the compiled file with
   node --require in the start script.
2. Add manual spans for every code path in row 6 of the inventory (queue consumers,
   custom clients), using tracer.startActiveSpan and recording exceptions.
3. Create src/logger.ts with pino: JSON in production, pino-pretty only in development,
   LOG_LEVEL defaulting to info, redaction for the fields in CLAUDE.md, and a mixin that adds
   trace_id and span_id from the active span.
4. Replace every console.log/warn/error call with the logger. Report the count before and after.
5. Install the latest versions and print the installed version of every new package.
6. Start the service, send one GET /health request, and print its trace ID and the log line
   that carries the same trace_id.
Do not add metrics, alerts or dashboards yet.

Check step 6 yourself against a real collector rather than trusting the transcript. A throwaway collector with the debug exporter prints every span it receives:

# otel-debug.yaml: a local collector that only prints what it receives
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
exporters:
  debug: { verbosity: detailed }
service:
  pipelines:
    traces: { receivers: [otlp], exporters: [debug] }

# Terminal 1: run the collector
docker run --rm -p 4317:4317 -p 4318:4318 \
  -v "$PWD/otel-debug.yaml:/etc/otelcol-contrib/config.yaml" otel/opentelemetry-collector-contrib

# Terminal 2: send a request, then search the collector output for the trace ID Claude printed
curl -s localhost:3000/health

The instrumentation is proven when the same trace ID appears in the collector output and in one JSON log line. If it appears in only one of them, the correlation is broken, and no later step can fix that.

Make every alert edit prove itself with a hook

Alert rules are small programs, and promtool test rules is their test runner. The Claude Code-specific part is making the check automatic: a PostToolUse hook runs the validator after every edit under observability/, and exit code 2 feeds the failure straight back to Claude, which fixes it on the next step. The mechanics are covered in hooks automation.

#!/usr/bin/env bash
# .claude/hooks/check-observability.sh (PostToolUse, matcher "Edit|Write")
set -uo pipefail
file=$(jq -r '.tool_input.file_path // empty')
dir=$CLAUDE_PROJECT_DIR/observability

case "$file" in
  "$dir"/alerts*.yml)
    if ! out=$(cd "$dir" && promtool check rules alerts.yml 2>&1 && promtool test rules alerts_test.yml 2>&1); then
      echo "Alert rules fail their checks. Fix alerts.yml, not the test file:" >&2
      tail -n 30 <<<"$out" >&2
      exit 2
    fi ;;
  "$dir"/alertmanager.yml)
    if ! out=$(amtool check-config "$file" 2>&1); then
      echo "Alertmanager config is invalid:" >&2; tail -n 20 <<<"$out" >&2; exit 2
    fi ;;
  "$dir"/dashboards/*.json)
    jq empty "$file" 2>/dev/null || { echo "Dashboard JSON does not parse: $file" >&2; exit 2; } ;;
esac
exit 0

Register the hook and the deny rule in .claude/settings.json. The deny rule is what stops an agent told to “make the tests pass” from weakening the assertion instead of fixing the rule; the reasoning is in protecting the oracle.

{
  "permissions": {
    "deny": ["Edit(./observability/alerts_test.yml)"]
  },
  "hooks": {
    "PostToolUse": [
      { "matcher": "Edit|Write",
        "hooks": [{ "type": "command", "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/check-observability.sh", "args": [], "timeout": 60 }] }
    ]
  }
}

A human writes alerts_test.yml, or Claude drafts it in a separate session before the deny rule is in place and the tech lead reviews it. It is short: a few synthetic series and the moments an alert must and must not fire. That file is where the intent lives, so it is the one file worth reading.

Copy-paste prompt: burn-rate alerts against a protected test file

Write Prometheus alert rules for a 99.9% availability SLO on this service.

1. List the real metric and label names first: query /api/v1/label/__name__/values on our
   Prometheus and show me the http_server_* names you found. Use only names from that list.
2. In observability/alerts.yml, write a multi-window burn-rate pair: fast (1h and 5m windows)
   routed as severity=critical, slow (6h and 30m windows) as severity=warning. Every rule gets
   summary and runbook_url.
3. observability/alerts_test.yml already exists and you cannot edit it. Make every test in it
   pass. The PostToolUse hook runs promtool after each edit; read its output and fix the rule.
4. Write observability/alertmanager.yml that routes critical to the pagerduty receiver and
   warning to the slack receiver, grouped by alertname and service, with an inhibit rule so a
   critical alert mutes the matching warning. Show amtool config routes test output for
   severity=critical and severity=warning.
If you believe a test is wrong, stop and explain why instead of working around it.

Expect one naming trap here. In @opentelemetry/instrumentation-http 0.222.0 (checked 2026-09-26), the Node HTTP instrumentation emits only the stable http.server.request.duration histogram, in seconds. Most training data uses the older http.server.duration or http_requests_total. Step 1 of the prompt exists for that reason; the details are in monitoring and observability with AI agents.

Check every dashboard panel through the Grafana MCP server

A dashboard with a wrong metric name renders “No data”, which looks like a quiet system. Give Claude read-only access to Grafana and have it run every panel’s query instead of clicking through them. Add the server at project scope so it loads only in this repository. mcp-grafana 1.6.0 on PyPI is the official Grafana Labs server (checked 2026-09-26); the wider catalogue is on observability MCP servers.

# Terminal: Viewer-role service-account token, writes disabled
claude mcp add grafana --scope project \
  -e GRAFANA_URL=https://grafana.example.com \
  -e 'GRAFANA_SERVICE_ACCOUNT_TOKEN=${GRAFANA_SERVICE_ACCOUNT_TOKEN}' \
  -- uvx mcp-grafana --disable-write

--scope project writes .mcp.json, which is committed. The single quotes matter: they store the literal ${GRAFANA_SERVICE_ACCOUNT_TOKEN}, which Claude Code expands from each developer’s environment at startup. With double quotes the shell expands the token first, and it lands in the committed file in plain text.

Copy-paste prompt: dashboard with a no-empty-panels check

Build observability/dashboards/service.json: request rate by http_route, 5xx ratio,
p50/p95/p99 latency with histogram_quantile over the duration histogram, error-budget burn
for the 99.9% SLO, and one panel per queue consumer from the inventory. Template the service
as $service.

Then, using the Grafana MCP server, run every panel's query for the last 1h with
service set to this service, and give me a table: panel title | query | series returned |
first value. Any panel with zero series is a failure: find the correct metric name through
the Grafana MCP server and fix the panel. Do not import or change anything in Grafana.

Produce the evidence with a verifier subagent

The session that wrote the configs is a weak judge of them. A custom subagent with no edit tools re-runs every check from a clean context and reports. Save it as .claude/agents/observability-verifier.md:

---
name: observability-verifier
description: Verifies observability changes. Use before opening a pull request that touches observability/, src/instrumentation.ts or src/logger.ts.
tools: Read, Grep, Glob, Bash
---

You verify, you never edit. Run and report, each with its exit code:
1. promtool check rules and promtool test rules in observability/.
2. amtool check-config observability/alertmanager.yml.
3. jq empty on every observability/dashboards/*.json.
4. A grep for console.log/warn/error under src/ (expect zero outside tests).
5. A grep for logger calls that pass req, req.body, req.headers or a whole user object.
6. git diff --exit-code origin/HEAD -- observability/alerts_test.yml (expect no diff).
Finish with a PASS or FAIL line per check and nothing else.

To produce the same report without an interactive session, run it headless and keep the JSON with the pull request. dontAsk denies anything your permission rules do not allow, so the run cannot drift into edits.

# Terminal, from the repository root (not a CI job holding production secrets)
claude -p "Use the observability-verifier subagent on this branch, wait for it to finish, and print its report." \
  --permission-mode dontAsk \
  --allowedTools "Bash(promtool *)" "Bash(amtool *)" "Bash(jq *)" "Bash(grep *)" "Bash(git diff *)" \
  --output-format json > observability-evidence.json

The deterministic checks (promtool, amtool) also belong in CI as required steps; the cross-tool page has a ready CI job for alert tests. The subagent adds the judgment calls a linter cannot make, such as a logger call that leaks a request body.

Who signs off on observability that Claude Code built?

The service owner signs off on evidence; the tech lead owns thresholds and routing. Neither reads the generated configs end to end. The pull request carries:

• The approved plan-mode inventory, including the row of paths auto-instrumentation misses.
• One trace ID visible in the collector output and in a JSON log line.
• The observability-evidence.json report with every check at PASS.
• The panel table from the dashboard prompt, with no empty panels.
• A diff that leaves alerts_test.yml untouched, or a separate reviewed commit that changes it.

The tech lead reads alerts_test.yml, the SLO target and the Alertmanager routes, because those encode intent: what counts as broken, and who gets woken up.

How do you monitor Claude Code itself?

Claude Code exports its own OpenTelemetry metrics and events. In version 2.1.283 the metric names include claude_code.session.count, claude_code.token.usage, claude_code.cost.usage, claude_code.lines_of_code.count, claude_code.commit.count, claude_code.pull_request.count and claude_code.code_edit_tool.decision. To confirm export works on your machine, print metrics to the terminal first:

# Terminal: a session.count datapoint should print within about 10 seconds
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_METRICS_EXPORTER=console
export OTEL_METRIC_EXPORT_INTERVAL=10000
claude

Switch OTEL_METRICS_EXPORTER to otlp and set OTEL_EXPORTER_OTLP_PROTOCOL and OTEL_EXPORTER_OTLP_ENDPOINT to point at your collector. The variable is CLAUDE_CODE_ENABLE_TELEMETRY; CLAUDE_CODE_TELEMETRY is not read (checked against 2.1.283). Team-wide rollout through managed settings, joining sessions to pull requests, and the cost-per-change dashboard belong to agent observability. Spend control is covered in cost control.

What breaks when Claude Code builds your observability?

Queries and alerts use metric names that do not exist. Panels show “No data” and promtool passes because the test series use the same invented names. Recovery: paste the output of curl -s localhost:9090/api/v1/label/__name__/values | jq -r '.data[]' | grep '^http_server' into the session and ask Claude to rewrite every query and test series under observability/ to those names, listing each change.

The hook never runs. Claude edits alerts.yml and nothing happens. Open /hooks in the session to see which hooks loaded and from which settings file. The usual causes are a script without the execute bit, a missing promtool on PATH, or exit 1 where the script should exit 2: only exit 2 feeds the output back to Claude.

Claude works around the protected test file. With the edit denied, it may copy the tests into a new file or loosen the rule until the old tests pass vacuously. Recovery: ask “List every file you created or changed under observability/ and explain each change to alerts.yml in terms of which test it fixed.” Reject anything that changes thresholds the tech lead did not approve. The deny rule covers only the file-edit tools: a Bash command such as sed -i or cp can still rewrite alerts_test.yml, which is why the verifier subagent’s check 6 diffs the file against the base branch.

Log volume or cost jumps after the logging migration. JSON lines are larger, and a health check that logs on every probe multiplies it. Ask: “Drop logging for GET /health and GET /ready, sample other 2xx request logs at 10%, and keep every warn and error. Show the change in log lines per minute on a local load run of 60 seconds.”

Personal data appears in logs. The logger logs whole request objects in places the inventory flagged. Recovery: run the verifier subagent’s check 5, then ask Claude to replace each flagged call with an explicit field list and extend the redaction paths. Treat the logs already shipped as an incident for your data-protection process, not a code fix.

Traces stop at a queue or a custom client. Auto-instrumentation did not cover row 6 of the inventory. Ask: “For each consumer and client in row 6, show where the traceparent is injected on send and extracted on receive, add what is missing, and prove it with one trace that spans producer and consumer in the collector output.”

Where to go next with Claude Code observability

Monitoring and observability with AI agents The full stack: collector, Prometheus, Loki and Tempo configs, metric naming and CI checks, for all three tools

Performance analysis with Claude Code Use the traces and latency histograms you just added to find and fix bottlenecks

AI-powered incident response Put these signals and the Grafana MCP server to work when the alert fires

Deployment automation with Claude Code Wire the burn-rate alerts into deploys for automated rollback