Skip to content
AI Developer Toolkit

Monitoring and observability

Monitoring and observability with AI agents means having Claude Code, Codex or Cursor instrument a service with OpenTelemetry, generate the collector, dashboard and alert configuration, and prove each piece with machine checks — promtool rule tests, config validators and live queries through the Sentry and Grafana MCP servers — so nobody has to read the generated YAML line by line.

Your checkout service throws intermittent 504s. Traces stop at the gateway span, the dashboard is green, and the alert that should have fired references a metric name your instrumentation stopped emitting two releases ago. An agent can wire the whole stack in an afternoon; this page is about making sure what it wires is true.

This page is for the developer who owns a service and the tech lead who signs off on its alerting. It covers observing your service. To observe the agents themselves — their traces, tool calls, cost and audit logs — see agent observability.

What an agent-built monitoring setup gives you

Section titled “What an agent-built monitoring setup gives you”
  • An OpenTelemetry bootstrap for a Node.js service on the current packages, with the metric names those packages actually emit.
  • Alert rules that ship with promtool unit tests, so “does this alert fire at 10% errors and stay quiet at 1%?” is a CI result, not a code-review opinion.
  • Validation commands for every generated config (Collector, Loki, Alertmanager) and a prompt that checks every dashboard panel returns data.
  • Read-only Sentry and Grafana MCP access, so the agent reads live metric names and errors instead of guessing them.
  • A recovery playbook for the five ways agent-built observability goes wrong.

In what order should you build service monitoring with an agent?

Section titled “In what order should you build service monitoring with an agent?”

Order matters because each layer is only as good as the one under it. An alert on a metric that does not exist never fires, and nothing reports that it never fires.

StepWhat the agent generatesWhat proves it without reading it
1. InstrumentOpenTelemetry SDK bootstrapA request’s trace ID found in Tempo; metric names listed from Prometheus
2. PipelineCollector, Prometheus, Loki, Tempo, Grafana configotelcol validate, loki -verify-config=true, promtool check config
3. AlertsPrometheus rules and Alertmanager routingpromtool test rules, amtool check-config
4. DashboardsGrafana dashboard JSONEvery panel query returns data (checked through the Grafana MCP)
5. Live accessSentry and Grafana MCP serversRead-only tokens; one incident question answered end to end

Do not let the agent skip ahead to dashboards. Dashboards built before step 1 is proven are built on guessed metric names.

Instrument a Node.js service with OpenTelemetry

Section titled “Instrument a Node.js service with OpenTelemetry”

The prompt is the same in all three tools; what differs is where you run it and how edits are approved. The prompt pins the current API because models trained on older tutorials reach for the removed Resource class and for metric names that no longer exist.

Run it from the repository root. Interactively, start claude and paste the prompt. For a scripted run, save the prompt to a file and use print mode. acceptEdits approves file edits only and denies any shell command your permission rules do not allow, so allow the three commands the prompt needs to install packages, start the service and call GET /health (flags checked on Claude Code 2.1.283):

Terminal window
claude -p "$(cat prompts/otel-bootstrap.md)" --permission-mode acceptEdits \
  --allowedTools "Bash(npm *)" "Bash(node *)" "Bash(curl *)" --output-format json

The JSON result carries the session ID and final message, which you can archive with the pull request.

The result should look close to this. Package versions current on 2026-09-26: @opentelemetry/sdk-node 0.222.0, @opentelemetry/resources 2.11.0, @opentelemetry/semantic-conventions 1.43.0.

// src/instrumentation.ts — compiled, then loaded with: node --require ./dist/instrumentation.js dist/server.js
// (ESM service: node --experimental-loader=@opentelemetry/instrumentation/hook.mjs --import ./dist/instrumentation.js dist/server.js)
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { resourceFromAttributes } from '@opentelemetry/resources';
import {
  ATTR_DEPLOYMENT_ENVIRONMENT_NAME,
  ATTR_SERVICE_NAME,
  ATTR_SERVICE_VERSION,
} from '@opentelemetry/semantic-conventions';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';


const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: process.env.SERVICE_NAME ?? 'checkout',
    [ATTR_SERVICE_VERSION]: process.env.SERVICE_VERSION ?? '0.0.0',
    [ATTR_DEPLOYMENT_ENVIRONMENT_NAME]: process.env.NODE_ENV ?? 'development',
  }),
  traceExporter: new OTLPTraceExporter(),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter(),
    exportIntervalMillis: 30_000,
  }),
  instrumentations: [
    getNodeAutoInstrumentations({ '@opentelemetry/instrumentation-fs': { enabled: false } }),
  ],
});


sdk.start();
process.on('SIGTERM', () => sdk.shutdown().finally(() => process.exit(0)));

The exporters read OTEL_EXPORTER_OTLP_ENDPOINT from the environment on their own. A plain node --require ./src/instrumentation.ts fails on a .ts file, which is why the prompt asks for the compiled file. An ESM service ("type": "module") needs more: --require does not hook ES module imports, so Express and http can run without producing a single span. Load the file with node --import ./dist/instrumentation.js and register the loader hook @opentelemetry/instrumentation/hook.mjs as the OpenTelemetry ESM guide describes, then confirm the setup by finding the request’s trace ID in Tempo.

Which HTTP metric names does current OpenTelemetry emit?

Section titled “Which HTTP metric names does current OpenTelemetry emit?”

This gotcha breaks agent-written alerts silently. From @opentelemetry/instrumentation-http 0.222.0 (checked 2026-09-26), the package README says it emits only the stable HTTP semantic conventions: the histogram http.server.request.duration, in seconds, with attributes such as http.route and http.response.status_code. Versions up to 0.221.0 emitted the old http.server.duration in milliseconds by default. Tutorials written before 0.222.0, and so models trained on them, use the old name.

After the OTLP-to-Prometheus translation, the new histogram usually appears as http_server_request_duration_seconds_bucket, _count and _sum, with labels http_route and http_response_status_code. “Usually” matters: the exact names depend on your Prometheus translation settings. So the rule is: the agent reads metric names from the running Prometheus before it writes a single query.

Terminal window
# Terminal: list what Prometheus actually stores
curl -s http://localhost:9090/api/v1/label/__name__/values | jq -r '.data[]' | grep '^http_server'

If that list is empty, stop. Nothing downstream can be right.

Stand up the collector, Prometheus, Loki, Tempo and Grafana

Section titled “Stand up the collector, Prometheus, Loki, Tempo and Grafana”

Ask for the Compose file (or Helm values) in one prompt, then validate each generated config with the component’s own checker before starting anything. Validators are faster than debugging a crash loop and they do not require you to read the file. Replace LOKI_VERSION with the 3.x tag you deploy.

Terminal window
# Terminal: validate the configs the agent generated
otelcol validate --config=otel-collector.yaml   # use your distribution's binary, e.g. otelcol-contrib
promtool check config prometheus.yml
docker run --rm -v "$PWD":/config grafana/loki:LOKI_VERSION -config.file=/config/loki.yaml -verify-config=true

Loki is where agents most often emit dead syntax. Loki 3.0 removed the shared_store settings for the TSDB and BoltDB shippers, and it enables structured metadata by default, which requires the active schema to use both the tsdb index and schema v13 (per Grafana’s Loki 3.0 upgrade guide). A generated config with boltdb-shipper, schema: v11 or shared_store fails validation. The current shape:

# loki.yaml (Loki 3.x): TSDB index, schema v13, no shared_store anywhere
schema_config:
  configs:
    - from: 2026-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h
storage_config:
  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache
  aws:
    s3: s3://s3.eu-central-1.amazonaws.com/checkout-logs
    s3forcepathstyle: true
compactor:
  working_directory: /loki/compactor

Generate alert rules you can test without reading them

Section titled “Generate alert rules you can test without reading them”

An alert rule is a small program, and promtool test rules is its unit-test runner. The agent writes the rule and the test together; the test feeds synthetic series and asserts when the alert fires and when it stays silent. You review the test cases — “fires at 10% errors, silent at 1%” — which are two lines of intent, not the PromQL.

alerts.yml
groups:
  - name: checkout-slo
    rules:
      - alert: CheckoutHighErrorRatio
        expr: |
          sum by (job) (rate(http_server_request_duration_seconds_count{job="checkout", http_response_status_code=~"5.."}[5m]))
            /
          sum by (job) (rate(http_server_request_duration_seconds_count{job="checkout"}[5m]))
            > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "5xx ratio above 5% on {{ $labels.job }}"
          runbook_url: "https://runbooks.example.com/checkout-5xx"
# alerts_test.yml — run with: promtool test rules alerts_test.yml
rule_files: [alerts.yml]
evaluation_interval: 1m
tests:
  # 10% of requests fail: the alert must fire once `for: 5m` has elapsed
  - interval: 1m
    input_series:
      - series: 'http_server_request_duration_seconds_count{job="checkout", http_route="/pay", http_response_status_code="500"}'
        values: '0+10x20'
      - series: 'http_server_request_duration_seconds_count{job="checkout", http_route="/pay", http_response_status_code="200"}'
        values: '0+90x20'
    alert_rule_test:
      - eval_time: 3m
        alertname: CheckoutHighErrorRatio
        exp_alerts: []
      - eval_time: 15m
        alertname: CheckoutHighErrorRatio
        exp_alerts:
          - exp_labels: { severity: critical, job: checkout }
            exp_annotations:
              summary: "5xx ratio above 5% on checkout"
              runbook_url: "https://runbooks.example.com/checkout-5xx"
  # 1% of requests fail: the alert must stay silent
  - interval: 1m
    input_series:
      - series: 'http_server_request_duration_seconds_count{job="checkout", http_route="/pay", http_response_status_code="500"}'
        values: '0+1x20'
      - series: 'http_server_request_duration_seconds_count{job="checkout", http_route="/pay", http_response_status_code="200"}'
        values: '0+99x20'
    alert_rule_test:
      - eval_time: 15m
        alertname: CheckoutHighErrorRatio
        exp_alerts: []

Keep the test file out of the agent’s reach when it later edits the rule, for the same reason you protect any test oracle: an agent told to “make the test pass” can weaken the assertion instead of fixing the rule. See protecting the oracle.

Alertmanager routing gets the same treatment: amtool check-config alertmanager.yml catches syntax and unknown receivers, and amtool config routes test --config.file=alertmanager.yml severity=critical service=checkout prints which receiver a given label set reaches.

Put the checks in CI

Section titled “Put the checks in CI”

The validators only protect you if they run on every change. A job like this makes “the alert fires at 10% errors” a required check on the pull request:

# .github/workflows/observability.yml (excerpt)
jobs:
  alert-tests:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - name: Test alert rules
        run: |
          docker run --rm -v "$PWD/observability:/w" -w /w --entrypoint promtool \
            prom/prometheus:PROMETHEUS_VERSION check rules alerts.yml
          docker run --rm -v "$PWD/observability:/w" -w /w --entrypoint promtool \
            prom/prometheus:PROMETHEUS_VERSION test rules alerts_test.yml

Replace PROMETHEUS_VERSION with the tag of the Prometheus you run in production, so the rules are tested by the same parser that loads them. The wider pipeline lives in CI/CD pipelines.

Check every dashboard panel returns data

Section titled “Check every dashboard panel returns data”

A generated dashboard fails silently: a wrong metric name renders as “No data”, which looks like a quiet system. Rather than clicking through panels, have the agent run every panel’s query through the Grafana MCP server and report the ones that return nothing.

The table is the evidence you attach to the pull request. A tech lead approves on “every panel returns series” plus a screenshot, not on the JSON.

Give the agent read access to production signals with MCP

Section titled “Give the agent read access to production signals with MCP”

Two vendor-official MCP servers turn the agent into an on-call reader: Sentry (issues, events, traces and Seer analysis) and Grafana (dashboards, Prometheus, Loki, incidents). Popularity, read 2026-09-26 from the MCP dossier: grafana/mcp-grafana has about 3.5k GitHub stars and getsentry/sentry-mcp about 862. Sentry is also in the Claude Code and Codex plugin marketplaces; Grafana ships Claude Code plugins (grafana-mcp, grafana-cloud-mcp). Datadog users have an official remote server at https://mcp.datadoghq.com/v1/mcp (US1; other sites change the host). The full catalogue is on observability MCP servers.

Terminal window
# Sentry: remote server with OAuth; authenticate with /mcp inside a session
claude mcp add --transport http sentry https://mcp.sentry.dev/mcp


# Grafana: read-only; use a Viewer-role service-account token
claude mcp add grafana -e GRAFANA_URL=https://grafana.example.com \
  -e GRAFANA_SERVICE_ACCOUNT_TOKEN="$GRAFANA_TOKEN" -- uvx mcp-grafana --disable-write

If your team uses Sentry, its skills complement the MCP server: sentry-instrument, sentry-debug-issue and sentry-otel-exporter-setup ship in getsentry/sentry-for-ai. Install them with claude plugin install sentry@claude-plugins-official; in Codex, with /plugins → search “sentry” (the entry is listed in openai-curated; the CLI form codex plugin add sentry@openai-curated needs a signed-in account, because the marketplace is provisioned at sign-in; checked on codex-cli 0.157.1); for Cursor, install Sentry’s Cursor plugin (getsentry/plugin-cursor); other agents can fetch the same skills from skills.sentry.dev. The getsentry/sentry-for-ai repository itself is the skill source, and its README says to install a client plugin rather than the repository (checked 2026-10-02). Reach for a skill when you want a repeatable procedure; reach for the MCP server when the agent needs live data.

Who signs off on agent-generated monitoring?

Section titled “Who signs off on agent-generated monitoring?”

The service owner signs off on the evidence; the tech lead owns the alerting policy. Neither reads the generated YAML end to end. The pull request that adds or changes monitoring carries:

  • The list of http_server_* metric names from the running Prometheus, and the trace ID of one request found in Tempo.
  • Green otelcol validate, promtool check config, loki -verify-config=true and amtool check-config output.
  • Green promtool test rules, with every alert covered by one must-fire and one must-stay-silent case.
  • The panel table from the dashboard prompt with no empty panels.
  • Confirmation that the MCP tokens are read-only.

The tech lead reviews the test cases and thresholds, because that is where the intent lives: the 5% threshold, the for: 5m delay and the SLO target.

What breaks when an agent builds your observability?

Section titled “What breaks when an agent builds your observability?”

Symptom: empty panels, alerts that never fire, promtool test rules green but production silent. Cause: the agent wrote http_server_duration or http_requests_total from memory, while the instrumentation emits http.server.request.duration. Recovery: run the metric-name curl above, then give the agent the list:

Here is the output of /api/v1/label/__name__/values filtered to http_server_*. Rewrite every
query in observability/ to use only these names, rerun promtool test rules with series named
the same way, and list each query you changed.

Tag each of these in your incident log with the class of agent failure behind it — most are “invented API” or “stale training data” — so you can see which control to add next. The classes are defined in the failure taxonomy.

Where to go next with monitoring and observability

Section titled “Where to go next with monitoring and observability”

Written by

See what's new on the platform.

Privacy Policy Terms of Service Cookie settings