Skip to content
AI Developer Toolkit

Find your level — free Buy the Toolkit Buy Toolkit Buy the Toolkit Buy Toolkit Open the guides Read
Navigation menu
Buy the Toolkit Find your level — free Buy the Toolkit Open the guides Sign in

Monitoring & Logging Patterns

PL

Level 3 How-to Updated

Monitoring and logging patterns are copy-paste recipes for having Claude Code, Codex, or Cursor write observability configuration: OpenTelemetry bootstrap code, golden-signal and SLO burn-rate alerts, Alertmanager routing, log pipelines that carry trace_id, and collector tail sampling. Each recipe ends in a machine check (promtool, amtool, or a config test), so nobody has to proofread PromQL.

This page is for developers and tech leads who own a service’s alerts. Your checkout API threw 500s for an hour last night while every dashboard stayed green, because the alerts watched CPU instead of the error rate. The one trace that would explain it was sampled away, and the logs from that minute carry no trace_id.

For the end-to-end build (instrumentation, the collector stack, dashboards, and sign-off), follow monitoring and observability. This page holds the individual recipes you reach for once that stack exists.

What these monitoring recipes give you

Section titled “What these monitoring recipes give you”
  • A golden-signals prompt that turns a four-line spec into latency, error, traffic, and saturation rules with a promtool test for each
  • A prompt that turns an SLO target into recording rules and a multi-window burn-rate alert pair, with promtool tests that prove when each alert fires
  • An Alertmanager routing prompt, plus the amtool commands that prove which receiver a given alert reaches
  • A log pipeline prompt that promotes trace_id to a top-level field and redacts PII, tested with --config.test_and_exit
  • A collector tail-sampling recipe that keeps every error trace and samples the rest
  • Two read-only audit prompts: one for alert-fatigue antipatterns in existing Prometheus rules or CloudWatch alarms, one for unbounded label cardinality
  • A migration prompt for monitoring code still on the AWS SDK v2 or the old OpenTelemetry metrics package

Which check proves each monitoring artifact?

Section titled “Which check proves each monitoring artifact?”

Agents write observability configuration quickly and get it subtly wrong: a metric name from old training data, a label that explodes cardinality, a burn-rate constant copied from a different SLO. Reading the YAML does not catch those. A validator and a behaviour test do. Give the agent the check together with the task, and ask it to paste the output.

Artifact the agent writesSyntax checkBehaviour check
Prometheus recording and alert rulespromtool check rules rules.ymlpromtool test rules rules_test.yml: synthetic series, “fires at X, silent at Y”
alertmanager.ymlamtool check-config alertmanager.ymlamtool config routes test --config.file=alertmanager.yml severity=critical prints the receiver
OpenTelemetry Collector configotelcol-contrib validate --config=collector.yamlthe debug exporter with verbosity: detailed shows spans arriving
Logstash pipelinebin/logstash -f logstash.conf --config.test_and_exitfeed one real log line and inspect the output event
Instrumentation codetype check and unit teststhe metric names exist in Prometheus: curl -s localhost:9090/api/v1/label/__name__/values

You review the behaviour check, not the configuration. “The fast-burn alert fires at 10% errors and stays silent at 0.3%” is two lines of intent; the PromQL behind it is the agent’s job.

Give the agent live telemetry through MCP first

Section titled “Give the agent live telemetry through MCP first”

An agent that cannot see your metrics invents thresholds and metric names. With the official Sentry and Grafana MCP servers connected, the agent reads the real metric list, the p95 for the last week, and the stack trace behind an error spike before it writes a rule. The URLs and packages are the same in every agent; only the command differs.

Terminal window
# Terminal, repo root. Sentry: hosted server, OAuth on first use (or /mcp in a session)
claude mcp add -s project --transport http sentry https://mcp.sentry.dev/mcp


# Grafana Labs' server from PyPI, read-only; the token file lives outside the repo
claude mcp add -s project grafana \
  -e GRAFANA_URL=https://grafana.example.com \
  -e 'GRAFANA_SERVICE_ACCOUNT_TOKEN_FILE=${HOME}/.config/grafana/mcp-token' \
  -- uvx mcp-grafana --disable-write

With Grafana connected, ask the agent to list the dashboards and data sources that already exist before it proposes a panel, so it extends them instead of duplicating them. Give the Grafana service account the Viewer role. GRAFANA_API_KEY is deprecated in mcp-grafana (1.6.0 on PyPI, checked 2026-09-26); use GRAFANA_SERVICE_ACCOUNT_TOKEN or the _FILE variant. Avoid lookalike packages such as @leval/mcp-grafana (a community port) or the deprecated @elastic/mcp-server-elasticsearch npm package; Elastic’s current server is documented in the elastic/mcp-server-elasticsearch repository. Scopes, token headers, and context cost for each server are on observability MCP servers.

A skill is the lighter option when you need the agent to know how to write good monitoring configuration, not what is firing right now. Seth Hobson’s wshobson/agents repository ships prometheus-configuration, slo-implementation, grafana-dashboards, and distributed-tracing skills in its observability-monitoring plugin:

Terminal window
# Any skills-compatible agent (Claude Code, Codex, Cursor, and others)
npx skills add wshobson/agents --skill slo-implementation


# Claude Code, inside a session: the whole plugin
/plugin marketplace add wshobson/agents
/plugin install observability-monitoring@claude-code-workflows

Use the skill for drafting rules from scratch, and the MCP server for tuning thresholds against production. For scale, checked 2026-09-26: wshobson/agents had about 40.0k GitHub stars and grafana/mcp-grafana about 3.5k. Read a third-party skill before you install it; see installing and managing skills.

Turn a golden-signals spec into alert rules

Section titled “Turn a golden-signals spec into alert rules”

Alert on what users feel. Of the four golden signals (latency, traffic, errors, and saturation), page on errors and latency, and send saturation to a ticket. The fastest way to a sane first rule set is a four-line spec in plain language; the agent writes the PromQL and the test, and you review the spec and the test output.

The latency and traffic rules are the two that agents get wrong most often, so they are the two to look at in the result:

# monitoring/rules/checkout-golden-signals.yml (excerpt)
groups:
  - name: checkout-golden-signals
    rules:
      - alert: CheckoutLatencyP95High
        expr: |
          histogram_quantile(0.95,
            sum by (le) (rate(http_server_request_duration_seconds_bucket{job="checkout-api"}[5m]))
          ) > 0.5
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "checkout-api p95 latency above 500 ms"
          runbook_url: "https://runbooks.example.com/checkout-latency"
      - alert: CheckoutTrafficDrop
        # The floor stops the rule paging every night when traffic is naturally low
        expr: |
          sum(rate(http_server_request_duration_seconds_count{job="checkout-api"}[5m]))
            < 0.5 * sum(rate(http_server_request_duration_seconds_count{job="checkout-api"}[1h] offset 5m))
          and
          sum(rate(http_server_request_duration_seconds_count{job="checkout-api"}[1h] offset 5m)) > 1
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "checkout-api traffic fell below half of the previous hour"
          runbook_url: "https://runbooks.example.com/checkout-traffic"

The static 5% error rule is the blunt version of the burn-rate pair in the next section. Once the burn-rate alerts are in place, delete the static rule or demote it to a ticket, so one incident does not page twice.

Write SLO burn-rate alerts that are tested, not trusted

Section titled “Write SLO burn-rate alerts that are tested, not trusted”

CPU and latency thresholds say something is off. A burn-rate alert says you are spending error budget faster than you can afford. The multi-window, multi-burn-rate pattern from Google’s SRE workbook fires only when a long and a short window agree, which keeps it quiet through a single bad scrape and lets it reset quickly after recovery.

First, confirm the metric names. From @opentelemetry/instrumentation-http 0.222.0 (checked 2026-09-26), Node.js services emit the stable histogram http.server.request.duration in seconds, which Prometheus usually stores as http_server_request_duration_seconds_bucket, _count, and _sum. Most tutorials and most model training data still use http_requests_total or http_request_duration_seconds. The details are in which HTTP metric names current OpenTelemetry emits.

The recording rules the agent produces should look like this. Every alert then reads a precomputed ratio instead of repeating the division:

# monitoring/rules/api-slo.yml (excerpt)
groups:
  - name: api-slo-recording
    rules:
      - record: job:slo_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m]))
          / sum by (job) (rate(http_server_request_duration_seconds_count[5m]))
      - record: job:slo_errors:ratio_rate1h
        expr: |
          sum by (job) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[1h]))
          / sum by (job) (rate(http_server_request_duration_seconds_count[1h]))
      # ratio_rate30m and ratio_rate6h follow the same shape
  - name: api-slo-alerts
    rules:
      # 99.5% SLO -> budget 0.005. 14.4x burn -> error ratio > 0.072.
      # At that rate a 30-day budget is gone in about 2.1 days (30 / 14.4).
      - alert: ApiErrorBudgetFastBurn
        expr: |
          job:slo_errors:ratio_rate1h{job="api"} > (14.4 * 0.005)
          and
          job:slo_errors:ratio_rate5m{job="api"} > (14.4 * 0.005)
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "api is burning error budget 14.4x faster than the SLO allows"
          runbook_url: "https://runbooks.example.com/api-error-budget"

A test file for these rules has the shape shown in generate alert rules you can test without reading them. Keep that test file out of the agent’s reach when it later edits the rule, so “make the test pass” cannot mean “weaken the test”.

Route alerts with Alertmanager and prove the routing tree

Section titled “Route alerts with Alertmanager and prove the routing tree”

Routing is where alert storms start: group_wait: 0s on every route turns one cluster outage into hundreds of pages. Describe the escalation policy in plain language and let amtool prove the result.

Terminal window
# Terminal: the checks you run, or have the agent run and paste
amtool check-config monitoring/alertmanager.yml
amtool config routes test --config.file=monitoring/alertmanager.yml \
  --verify.receivers=pagerduty-oncall severity=critical service=checkout

--verify.receivers makes the command exit non-zero when the label set reaches a different receiver, so the same line works as a CI step.

Correlate logs with traces in the log pipeline

Section titled “Correlate logs with traces in the log pipeline”

The most valuable logging change is making every log line carry trace_id. Then you move from a Grafana span to the surrounding logs in one click. Field names are where agent-written pipelines break, so the prompt starts from one real log line.

The part to check is the correlation block. The field paths must match what your OpenTelemetry log bridge or logger emits:

filter {
  if [message] =~ /^\{/ {
    json { source => "message" target => "parsed" }
    mutate {
      rename => {
        "[parsed][trace_id]" => "trace_id"
        "[parsed][span_id]"  => "span_id"
        "[parsed][level]"    => "log_level"
      }
    }
  }
  if [environment] == "production" and [log_level] == "DEBUG" { drop {} }
}

On Loki instead of Elasticsearch, the prompt transfers: ask for a Grafana Alloy or Promtail configuration that extracts trace_id as structured metadata, not as a label. A label per trace ID is the same cardinality explosion described below. For the application side, structured and redacted logging with request correlation, see logging patterns.

Keep every error trace with tail sampling

Section titled “Keep every error trace with tail sampling”

Head sampling decides at the first span, before anyone knows whether the request failed, so the trace you need is often gone. Tail sampling decides after the trace completes: keep all errors and slow requests, and sample the rest.

# monitoring/otel-collector.yaml (excerpt)
processors:
  tail_sampling:
    decision_wait: 30s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Instrument a service without the deprecated exporters

Section titled “Instrument a service without the deprecated exporters”

Dashboards are downstream of instrumentation. Ask for OTLP explicitly. Older model knowledge reaches for @opentelemetry/exporter-jaeger, whose README marks it deprecated with support ended in March 2024 (npm 2.11.0, checked 2026-09-26); Jaeger accepts OTLP directly.

// src/tracing.js — node --require ./src/tracing.js server.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-proto');
const { resourceFromAttributes } = require('@opentelemetry/resources');
const {
  ATTR_SERVICE_VERSION,
  ATTR_DEPLOYMENT_ENVIRONMENT_NAME,
} = require('@opentelemetry/semantic-conventions');


const sdk = new NodeSDK({
  // service.name comes from OTEL_SERVICE_NAME
  resource: resourceFromAttributes({
    [ATTR_SERVICE_VERSION]: process.env.SERVICE_VERSION ?? '0.0.0',
    [ATTR_DEPLOYMENT_ENVIRONMENT_NAME]: process.env.ENVIRONMENT ?? 'production',
  }),
  // Reads OTEL_EXPORTER_OTLP_ENDPOINT; OTLP/HTTP is port 4318, gRPC is 4317
  traceExporter: new OTLPTraceExporter(),
  instrumentations: [getNodeAutoInstrumentations({
    '@opentelemetry/instrumentation-fs': { enabled: false },
  })],
});


sdk.start();
process.on('SIGTERM', () => sdk.shutdown().finally(() => process.exit(0)));

deployment.environment.name is the stable attribute in @opentelemetry/semantic-conventions 1.43.0; the older deployment.environment that many examples use is not. The collector stack this code exports to is in monitoring and observability.

Audit existing alerts and metrics with a read-only agent

Section titled “Audit existing alerts and metrics with a read-only agent”

Most teams do not lack alerts; they have the wrong ones. An audit is a good first agent job because it needs no write access: the agent reads rule files or an alarm export and reports.

How you run the audit differs by tool:

Run it headless with only read tools. In CI, add the isolation flags so a pull request cannot inject hooks, settings, or MCP servers into the job:

Terminal window
claude -p "$(cat prompts/alert-audit.md)" \
  --bare --setting-sources "" --strict-mcp-config \
  --tools "Read,Grep,Glob" --output-format json > audit.json

--bare never reads OAuth credentials or the keychain, connects only the MCP servers named on the command line (from v2.1.286), and authenticates with ANTHROPIC_API_KEY (or an apiKeyHelper passed with --settings), so read the key from your CI secret store. Fail the job when audit.json reports findings. The rules for agents in pipelines are on CI/CD pipeline recipes.

For cardinality, give the agent the numbers instead of asking it to guess. Prometheus reports its heaviest metrics directly:

Terminal window
# Terminal: top metrics by series count, and total head series
curl -s localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:10]'
curl -s 'localhost:9090/api/v1/query?query=prometheus_tsdb_head_series' | jq '.data.result[0].value[1]'

Paste both outputs into the audit prompt, and ask for a metric_relabel_configs rule that drops the offending label.

Migrate monitoring code off end-of-support SDKs

Section titled “Migrate monitoring code off end-of-support SDKs”

Monitoring code is often years old. Two migrations come up constantly, and both are mechanical enough for an agent. The aws-sdk npm package (v2) is deprecated on npm with the notice that it “has reached end-of-support, and no longer receives updates” (2.1693.0, checked 2026-09-26). @opentelemetry/metrics stopped at 0.24.0 and was renamed twice; the current package is @opentelemetry/sdk-metrics (2.11.0, checked 2026-09-26), where metric readers are passed to the MeterProvider constructor.

// src/monitoring/alarms.mjs — the migrated shape
import { CloudWatchClient, PutMetricAlarmCommand } from '@aws-sdk/client-cloudwatch';
import { MeterProvider } from '@opentelemetry/sdk-metrics';
import { PrometheusExporter } from '@opentelemetry/exporter-prometheus';


const cw = new CloudWatchClient({});
await cw.send(new PutMetricAlarmCommand({
  AlarmName: 'checkout-api-high-5xx',
  Namespace: 'AWS/ApplicationELB',
  MetricName: 'HTTPCode_Target_5XX_Count',
  Statistic: 'Sum',
  Period: 300,
  EvaluationPeriods: 3,
  DatapointsToAlarm: 2,
  Threshold: 10,
  ComparisonOperator: 'GreaterThanThreshold',
  TreatMissingData: 'notBreaching',
  AlarmActions: [process.env.SNS_TOPIC_ARN],
  OKActions: [process.env.SNS_TOPIC_ARN], // without it the alarm never reports recovery
}));


// PrometheusExporter is a MetricReader; it serves /metrics on port 9464 by default
const meterProvider = new MeterProvider({ readers: [new PrometheusExporter()] });
export const meter = meterProvider.getMeter('checkout-api');

The type checker catches most leftovers, such as a v2-only option name. Paste the compiler error back to the agent rather than describing it.

Who signs off on agent-written monitoring?

Section titled “Who signs off on agent-written monitoring?”

The agent writes the configuration and runs the checks; people approve intent. The service owner approves the SLO target and the test cases (“fires at 7.2% errors, silent at 0.3%”). The on-call lead approves routing, confirmed by the amtool output. A pull request that changes rules or routing without the matching test output in its description is not ready for review. What else that pull request must prove is on the evidence bundle.

What breaks when agents write monitoring and logging config?

Section titled “What breaks when agents write monitoring and logging config?”

  1. An alert storm follows a missing for: clause. A rule without for: fires on one bad scrape. Recover: grep the rule files for alerts with an expr and no for:, add a window, and add a promtool test with a one-sample spike that must stay silent.

  2. A latency panel or alert reads NaN. histogram_quantile is running over a series that is not a _bucket histogram, or le is missing from the by (...) clause. Recover: have the agent list the real metric names from Prometheus and rewrite the query from that list.

  3. An alert never fires because the metric does not exist. The rule uses http_requests_total or http_request_duration_seconds, but current OpenTelemetry emits http_server_request_duration_seconds_*. Recover: promtool test rules passes against synthetic series, so also run each alert expression once against the real Prometheus; an empty result means a wrong name.

  4. Prometheus runs out of memory, or scrapes slow down. A label the agent added, often user_id or a raw URL path, multiplied the series count. Recover: find the metric with /api/v1/status/tsdb, drop the label with metric_relabel_configs, and fix the instrumentation so the label becomes a bounded route template.

  5. Spans never arrive. The exporter protocol and port do not match the receiver: OTLP/HTTP is 4318, OTLP/gRPC is 4317. Recover: add the debug exporter with verbosity: detailed to the collector and confirm spans arrive before you blame the backend.

  6. Logs do not link to traces. The pipeline expects trace_id at the top level, but the logger nests it. Recover: log one raw event, give it to the agent, and have it fix the rename paths from that sample.

  7. A CloudWatch alarm looks stuck firing. It has AlarmActions but no OKActions, so nobody hears about recovery. Recover: pair every alarm action with an OK action; the audit prompt above finds the rest.

  8. A traffic-drop alert pages every night, or one incident pages twice. A relative traffic rule with no volume floor fires whenever traffic is naturally low, and a static error-ratio rule next to a burn-rate pair pages for the same outage twice. Recover: add the floor shown in the golden-signals example, and keep only the burn-rate pair paging for errors.

  9. The thresholds are invented. Without MCP access to real data, the agent produces confident numbers that do not match your traffic. Recover: connect Grafana read-only, have the agent quote last week’s p95 and error ratio, and set thresholds from those numbers.

Where to go next with monitoring and logging

Section titled “Where to go next with monitoring and logging”
Monitoring and observability The end-to-end workflow: instrumentation, the collector stack, dashboards and sign-off.
Observability MCP servers Sentry, Grafana, Datadog and PostHog MCP, read-only, with scopes and context cost.
Incident response What to do with the alert once it pages someone.
Debugging patterns Turn the telemetry behind an alert into a root cause.
Recovery patterns Automate the runbook an alert links to.
CI/CD pipeline recipes Run promtool and amtool on every pull request, safely.

Did this guide help you?

One tap, no sign-in.

not at all a lot
All 577 guides Free resources: Compare tools Glossary By role

Written by

See what's new on the platform.

Privacy Policy Terms of Service Cookie settings llms.txt pricing.md RSS