# Observing the agents: traces, tool calls, cost and audit logs

URL: https://developertoolkit.ai/en/workflows/operate/agent-observability/
Description: Export agent telemetry with OpenTelemetry, join runs to pull requests and incidents, build a run-success and cost-per-change dashboard, and set audit retention.

Agent observability means exporting each coding agent's telemetry (sessions, model requests, tool calls, permission decisions, and cost) through OpenTelemetry, stamping every run with IDs that join it to its pull request and to later incidents, and keeping audit events longer than analytics. Claude Code and Codex export OpenTelemetry natively; for Cursor, hooks and commit provenance fill the gap.

This page is for the **developer** who runs agent loops in CI, the **tech lead** who owns the loop's autonomy, and the **CTO** who signs the budget and the audit policy. The situation it solves: an agent-written change caused a production incident on Tuesday, and on Wednesday nobody can say which session wrote it, what the agent ran, who approved the tool calls, or what that loop costs per change that survives. You have logs for your services and none for the things writing them.

## What you get from agent observability done this way

- A collector with a pseudonymized analytics pipeline and an identified audit pipeline.
- Export settings for Claude Code, Codex, and the verified path for Cursor.
- Four join keys that lead from a commit to its pull request, session, and tool calls.
- An eight-panel dashboard with its SQL, a retention template, and three copy-paste prompts.

## Why do agent runs need their own telemetry?

Your service telemetry answers "what did the system do?". Agent telemetry answers a different question: "what did the thing that changed the system do, on whose authority, and at what cost?". Faros AI's *AI Engineering Report 2026: The Acceleration Whiplash* (April 2026, two years of telemetry from 22,000 developers on Faros's own platform) reports median time in review up 441.5% and incidents per pull request up 242.7%. When reviewers cannot read everything, evidence about the run has to come from telemetry.

Three things make agent telemetry different:

- **The unit is a run, not a request.** One run is a session of many model calls and tool calls, sometimes with subagents. Success is decided days later, when the pull request merges and survives.
- **Identity is a developer's.** Claude Code attributes every tool call to the developer who started the session; it does not act under a separate service account (Claude Code monitoring docs, checked 2026-09-26). The same data is both an audit trail and a potential surveillance tool.
- **Content is sensitive by default.** Prompts, tool arguments, and model responses can contain secrets and customer data. Both tools keep content out of telemetry unless you opt in.

## What does each tool emit?

Configure each tool separately and normalize afterwards.

| Concept | Claude Code 2.1.283 | Codex CLI 0.157.1 | Cursor |
| --- | --- | --- | --- |
| Export mechanism | OTel metrics and log events; traces in beta | OTel logs, traces, and metrics from the `[otel]` table | No OTel export verified on 2026-09-26; use hooks and provenance tools |
| Session or run ID | `session.id` (OTel); `session_id` in `claude -p --output-format json` | `conversation.id` (OTel); `thread_id` in `codex exec --json` | Cloud Agents API run IDs and `statusChange` webhooks (verified 2026-08-28) |
| Cost | `claude_code.cost.usage` (USD); `total_cost_usd` per headless run | `codex.turn.cost_microusd` metric; `codex.turn_cost` event with `usage.estimated_usd` | Not verified |
| Tokens | `claude_code.token.usage` by `type`, `model`, `effort` | `codex.turn.token_usage`; `usage` on each `turn.completed` | Not verified |
| Tool calls | `claude_code.tool_result` (`tool_name`, `success`, `duration_ms`) | `codex.tool_result`; `codex.tool.call` metric | Hooks (JSON over stdio); event names not verified |
| Permission decisions | `claude_code.tool_decision`, `claude_code.permission_mode_changed` | `codex.tool_decision`, `codex.sandbox_outcome` | Not verified |
| Link to a commit | `vcs.ref.head.revision` on the `tool_result` of a successful `git commit` (needs `OTEL_LOG_TOOL_DETAILS=1`) | None; record the `thread_id` in the pull request | `Entire-Checkpoint` commit trailer from Entire CLI |
| Vendor dashboard | `claude.ai/analytics/claude-code` (Team, Enterprise); `platform.claude.com/claude-code` (Console) | Enterprise analytics dashboard (secondary: OpenAI help pages seen only in search extracts) | Not verified on 2026-09-26 (cursor.com unreachable) |

Sources: Claude Code [monitoring](https://code.claude.com/docs/en/monitoring-usage) and [analytics](https://code.claude.com/docs/en/analytics) docs; Codex source at tag `rust-v0.157.1` (`codex-rs/otel`, `codex-rs/config/src/types.rs`, `codex-rs/exec/src/exec_events.rs`), all read 2026-09-26. cursor.com was unreachable on 2026-09-26, so Cursor's analytics and audit endpoints are not named here.

**Where the OpenTelemetry GenAI conventions stand**

The shared vocabulary for agent spans (`invoke_agent`, `execute_tool`, `plan`) and for MCP (`mcp.method.name`, `gen_ai.tool.name`) lives in the `open-telemetry/semantic-conventions-genai` repository, which had no release on 2026-09-26; both documents carry **Status: Development**. Claude Code adds a few GenAI attributes, such as `gen_ai.request.model`, as aliases on its spans; Codex uses `codex.*` names. Build your dashboard on a small normalized view (step 5) rather than on either vendor's names, and remap when the conventions stabilize.

## Set up agent observability, step by step

1. **Write the measurement charter before you collect anything.** One paragraph, signed by the CTO: which questions the data answers (run success, cost per accepted change, audit), who sees identities (the security team, through the SIEM), who sees pseudonyms (everyone else), and that no metric is used to rate an individual. [The metrics frameworks page](/en/org/metrics-frameworks/) explains why PR counts and token counts become gamed targets, and [privacy and data handling](/en/org/privacy-data-handling/) covers vendor data terms and retention. Telemetry that engineers believe is a surveillance tool gets switched off in their shells.

2. **Stand up one OpenTelemetry Collector with split pipelines.** Every tool sends OTLP to the same collector. For the analytics backends the collector replaces the email with a keyed hash, deletes the other user identifiers, and drops tool arguments; the SIEM gets the unmodified audit stream:

    ```yaml
    # otel-collector.yaml (OpenTelemetry Collector contrib distribution)
    receivers:
      otlp:
        protocols:
          grpc: { endpoint: 0.0.0.0:4317 }
          http: { endpoint: 0.0.0.0:4318 }

    processors:
      batch: {}
      # Keyed hash: PSEUDONYM_SALT is a secret only the collector holds.
      transform/pseudonymize:
        error_mode: ignore
        log_statements:
          - context: log
            statements:
              - set(attributes["user.email"], SHA256(Concat([attributes["user.email"], "${env:PSEUDONYM_SALT}"], ""))) where attributes["user.email"] != nil
        metric_statements:
          - context: datapoint
            statements:
              - set(attributes["user.email"], SHA256(Concat([attributes["user.email"], "${env:PSEUDONYM_SALT}"], ""))) where attributes["user.email"] != nil
      attributes/drop-ids:
        actions:
          - key: user.account_uuid
            action: delete
          - key: user.account_id
            action: delete
          - key: user.id
            action: delete
          - key: enduser.id
            action: delete
      attributes/strip-content:
        actions:
          - key: tool_parameters
            action: delete
          - key: tool_input
            action: delete
          - key: error
            action: delete

    exporters:
      prometheus:
        endpoint: 0.0.0.0:8889
      otlphttp/analytics:
        endpoint: https://logs.internal.example.com/otlp
      otlphttp/siem:
        endpoint: https://siem.internal.example.com:4318

    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [transform/pseudonymize, attributes/drop-ids, batch]
          exporters: [prometheus]
        logs/analytics:
          receivers: [otlp]
          processors: [transform/pseudonymize, attributes/drop-ids, attributes/strip-content, batch]
          exporters: [otlphttp/analytics]
        logs/audit:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp/siem]
    ```

    Claude Code attaches `user.email`, `user.account_uuid`, `user.account_id` (the admin-API ID), and `user.id` (a persistent per-install ID) to every metric and event; hashing only the email leaves three ways back to each person. The attributes processor's `hash` action is unsalted, so a list of staff emails reverses it. `logs.internal.example.com` and `siem.internal.example.com` stand for your log store and your SIEM's OTLP receiver.

3. **Turn on export in each tool.** Developer machines get it through managed configuration; CI jobs set it in the job.

      **Claude Code**

        Put the exporter settings in managed settings. Claude Code ignores OpenTelemetry exporter variables in a repository's `.claude/settings.json` and `.claude/settings.local.json`, so a repository cannot turn telemetry on, redirect it, or capture content. When managed settings set `OTEL_EXPORTER_OTLP_ENDPOINT`, Claude Code removes developer-set per-signal endpoints at startup (v2.1.217 and later).

        ```json
        {
          "env": {
            "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
            "OTEL_METRICS_EXPORTER": "otlp",
            "OTEL_LOGS_EXPORTER": "otlp",
            "OTEL_EXPORTER_OTLP_PROTOCOL": "grpc",
            "OTEL_EXPORTER_OTLP_ENDPOINT": "https://otel-collector.internal.example.com:4317",
            "OTEL_METRICS_INCLUDE_REPOSITORY": "true",
            "OTEL_LOG_TOOL_DETAILS": "1"
          }
        }
        ```

        `OTEL_LOG_TOOL_DETAILS=1` adds Bash commands, MCP server and tool names, and tool input to events. It is what makes MCP activity auditable and what puts the commit SHA on `git commit` tool results; it is also where secrets typed on a command line end up, which is why the collector in step 2 strips those attributes from everything except the SIEM stream, and why the endpoint must be TLS (`https://`) or reachable only over a private network. Prompt text stays off unless you set `OTEL_LOG_USER_PROMPTS=1`.

        For a headless CI run, set the same variables on the job, and give the run its own session ID so the pull request can carry it:

        ```yaml
        # .github/workflows/agent-issue-to-pr.yml
        name: agent-issue-to-pr
        on:
          workflow_dispatch:
            inputs:
              issue:
                description: Issue number to implement
                required: true
                type: number
        permissions: {}
        env:
          ISSUE: ${{ inputs.issue }}
          BRANCH: agent/issue-${{ inputs.issue }}
        jobs:
          issue-to-pr:
            # Runs the agent. Read-only token: nothing here can push.
            runs-on: ubuntu-latest
            timeout-minutes: 30
            permissions:
              contents: read
              issues: read
            outputs:
              session_id: ${{ steps.sid.outputs.id }}
            env:
              CLAUDE_CODE_ENABLE_TELEMETRY: "1"
              OTEL_METRICS_EXPORTER: otlp
              OTEL_LOGS_EXPORTER: otlp
              OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
              OTEL_EXPORTER_OTLP_ENDPOINT: ${{ secrets.OTEL_ENDPOINT }}
              OTEL_EXPORTER_OTLP_HEADERS: ${{ secrets.OTEL_HEADERS }}
              OTEL_RESOURCE_ATTRIBUTES: ci.run_id=${{ github.run_id }},loop.name=issue-to-pr
              OTEL_METRICS_INCLUDE_RESOURCE_ATTRIBUTES: "false"
            steps:
              - uses: actions/checkout@v7
                with:
                  persist-credentials: false
              - uses: actions/setup-node@v7
                with:
                  node-version: 22
              - name: Install Claude Code
                run: npm install -g @anthropic-ai/claude-code
              - name: Prepare the branch and the prompt
                env:
                  GH_TOKEN: ${{ github.token }}
                run: |
                  git config user.name "issue-to-pr agent"
                  git config user.email "issue-to-pr-agent@users.noreply.github.com"
                  git switch -c "$BRANCH"
                  { cat .github/prompts/issue-to-pr.md; echo
                    gh issue view "$ISSUE" --json number,title,body \
                      --jq '"Issue #\(.number): \(.title)\n\n\(.body)"'
                  } > "$RUNNER_TEMP/prompt.md"
              - name: Pick a session ID
                id: sid
                run: echo "id=$(cat /proc/sys/kernel/random/uuid)" >> "$GITHUB_OUTPUT"
              - name: Run the agent
                env:
                  ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
                  SID: ${{ steps.sid.outputs.id }}
                run: |
                  status=0
                  claude -p "$(cat "$RUNNER_TEMP/prompt.md")" \
                    --session-id "$SID" --output-format json \
                    --max-budget-usd 10 --permission-mode dontAsk \
                    --allowedTools "Read,Grep,Glob,Edit,Write,Bash(npm test *),Bash(git commit *)" \
                    > "$RUNNER_TEMP/run.json" || status=$?
                  jq -c '{session_id, is_error, subtype, num_turns, total_cost_usd, duration_ms}' \
                    "$RUNNER_TEMP/run.json" >> "$RUNNER_TEMP/runs.jsonl"
                  exit "$status"
              - name: Keep the run record
                if: always()
                uses: actions/upload-artifact@v7
                with:
                  name: agent-run-${{ github.run_id }}
                  path: ${{ runner.temp }}/runs.jsonl
                  if-no-files-found: ignore
              - name: Bundle the agent's commits
                env:
                  BASE: ${{ github.sha }}
                run: git bundle create "$RUNNER_TEMP/agent.bundle" "$BRANCH" "^$BASE"
              - uses: actions/upload-artifact@v7
                with:
                  name: agent-commits-${{ github.run_id }}
                  path: ${{ runner.temp }}/agent.bundle
                  if-no-files-found: error

          open-pr:
            # Holds the write token. Runs no code from the agent's commits.
            needs: issue-to-pr
            runs-on: ubuntu-latest
            timeout-minutes: 5
            permissions:
              contents: write
              pull-requests: write
            steps:
              - uses: actions/checkout@v7
                with:
                  persist-credentials: false
              - uses: actions/download-artifact@v8
                with:
                  name: agent-commits-${{ github.run_id }}
                  path: ${{ runner.temp }}
              - name: Push and open the pull request
                env:
                  GH_TOKEN: ${{ github.token }}
                  AGENT_SESSION_ID: ${{ needs.issue-to-pr.outputs.session_id }}
                run: |
                  git fetch "$RUNNER_TEMP/agent.bundle" "$BRANCH:$BRANCH"
                  gh auth setup-git
                  git push origin "refs/heads/$BRANCH"
                  gh pr create --head "$BRANCH" --title "Fix #$ISSUE (agent)" \
                    --body "Closes #$ISSUE. Agent session: $AGENT_SESSION_ID"
        ```

        `OTEL_METRICS_INCLUDE_RESOURCE_ATTRIBUTES=false` keeps `ci.run_id` on events but off metric labels, where it would create one time series per CI run. The `subtype` field tells you why a run stopped: `success`, `error_max_turns`, `error_max_budget_usd`, `error_during_execution`, or `error_max_structured_output_retries`. The run is started by hand (`workflow_dispatch` needs write access to the repository) and checks out the ref chosen at dispatch, which is the default branch unless the person starting it picks another in "Use workflow from" or passes `gh workflow run --ref`. Steps inside the agent job are not isolated from each other: `Edit`, `Write` and `Bash(npm test *)` let the agent rewrite the test script and run any code it likes, the issue text it works from is untrusted, and that code can read `ANTHROPIC_API_KEY` and the OTLP headers and write to `$GITHUB_ENV` or `$GITHUB_PATH`, which every later step of the same job inherits. That is why the agent job holds only a read-only `GITHUB_TOKEN`, and the push lives in a separate `open-pr` job: it receives the commits as a git bundle artifact, gets the session ID from a step that ran before the agent, and runs no code from those commits. If the agent made no commits, `git bundle create` refuses an empty bundle and the run stops before `open-pr`. Read the issue before you dispatch the run; for workflows triggered by untrusted contributors, follow the [CI security rules](/en/workflows/ship/ci-cd-pipelines/) first. `gh pr create` with `GITHUB_TOKEN` works only when "Allow GitHub Actions to create and approve pull requests" is enabled in the repository's or organization's Actions settings, and a pull request opened with that token does not trigger `pull_request` workflows, so the agent's PR gets no CI run. If CI must run on it, open the PR with a GitHub App token or a fine-grained personal access token instead. The run record is written even when the agent stops with an error, so failed runs reach `runs.jsonl` too.

        Traces are beta: add `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1` and `OTEL_TRACES_EXPORTER=otlp`. In `claude -p` and Agent SDK sessions, Claude Code reads `TRACEPARENT` from its environment, so its `claude_code.interaction` spans nest under your CI job's trace.

      **Codex**

        Codex reads an `[otel]` table from `config.toml`. In 0.157.1 the log and trace exporters default to `none`, and `metrics_exporter` defaults to `statsig`, a Codex-internal metrics destination; set all three explicitly so metrics go where your policy says.

        ```toml
        # ~/.codex/config.toml (or written by the CI job at run time)
        [otel]
        environment = "ci"
        log_user_prompt = false
        exporter = { otlp-http = { endpoint = "https://otel-collector.internal.example.com:4318/v1/logs", protocol = "binary" } }
        trace_exporter = { otlp-http = { endpoint = "https://otel-collector.internal.example.com:4318/v1/traces", protocol = "binary" } }
        metrics_exporter = { otlp-http = { endpoint = "https://otel-collector.internal.example.com:4318/v1/metrics", protocol = "binary" } }

        [otel.span_attributes]
        "loop.name" = "issue-to-pr"
        ```

        Collector auth headers go in plain text in an inline `headers = { ... }` table per exporter, so write the file in the CI job from a secret and never commit it. Codex's main branch adds `log_agent_responses` and `log_guardian_assessments` opt-ins after 0.157.1; they are not in 0.157.1 (checked 2026-09-26), so do not copy them into a config for that release.

        For headless runs, `codex exec --json` prints JSONL events. The first is `thread.started` with the `thread_id`, which is the same thread ID the OTel events carry as `conversation.id`. `turn.completed` carries token usage and `turn.failed` or `error` marks a failed run:

        ```bash
        # CI step (Codex CLI 0.157.1), with OPENAI_API_KEY from a secret
        codex exec --json --sandbox workspace-write \
          "$(cat .github/prompts/issue-to-pr.md)" > events.jsonl
        jq -c -s '{
          thread_id: (map(select(.type == "thread.started"))[0].thread_id),
          failed: any(.[]; .type == "turn.failed" or .type == "error"),
          output_tokens: (map(select(.type == "turn.completed") | .usage.output_tokens) | add)
        }' events.jsonl >> runs.jsonl
        ```

        Run one smoke test per CI image and confirm the `codex.*` events reach the collector from `codex exec`, not only from interactive sessions, before you build panels on them (an open report, openai/codex#12913, says `codex exec` emits no OTel metrics; status unverified on 2026-09-26). If the metrics are absent, build Codex cost panels on the `output_tokens` usage recorded in `runs.jsonl` instead.

      **Cursor**

        On 2026-09-26 this page could verify no OpenTelemetry export for Cursor, so the verified path is provenance plus the cloud API:

        - **Hooks.** Cursor runs hooks that receive agent events as JSON over stdio (verified 2026-08-28). A hook can append each event to a log file your collector tails.
        - **Entire CLI** (`entireio/cli`, v0.11.3, 5.1k GitHub stars on 2026-09-26) installs git hooks plus agent hooks in `.cursor/hooks.json`. When the agent commits, it links prompts, transcript, tool calls, files touched, and tokens to the commit through an `Entire-Checkpoint:` trailer:

          ```bash
          # Terminal, in the repository (install first: brew install --cask entireio/tap/entire)
          entire enable --agent cursor
          entire status
          ```

          Entire stores transcripts in your git repository and redacts secrets only partially. On a public repository, point it at a private one with `--checkpoint-remote`.
        - **Cloud Agents.** Runs started through the Cloud Agents API report `statusChange` webhooks (verified 2026-08-28). Record the run ID and final status in `runs.jsonl` exactly as the other tabs do.

4. **Stamp every run with four join keys, and write them into the pull request.** Telemetry you cannot join to outcomes only tells you what you spent. Each run carries:

    | Key | Claude Code | Codex | Cursor | Where it lands |
    | --- | --- | --- | --- | --- |
    | Session | `--session-id` UUID | `thread_id` | Entire checkpoint ID or cloud run ID | Evidence bundle `provenance.session` |
    | CI run | `OTEL_RESOURCE_ATTRIBUTES` `ci.run_id` | `[otel.span_attributes]` | `runs.jsonl` | `runs.jsonl`, events |
    | Loop | `loop.name` resource attribute | `loop.name` span attribute | `runs.jsonl` | Dashboard grouping |
    | Task | Issue URL in the prompt and the bundle | Same | Same | Evidence bundle `provenance.task` |

    The pull request's [evidence bundle](/en/workflows/ship/evidence-bundle/) already has a `provenance.session` field; the CI job fills it from `AGENT_SESSION_ID` or the `thread_id`. Join on the session, not on the commit SHA: a squash merge creates a new SHA, so the SHA recorded at `git commit` time never appears on the default branch. On Team and Enterprise plans with the GitHub app, Claude Code's analytics also labels merged pull requests `claude-code-assisted`, a cross-check that is unavailable under Zero Data Retention.

5. **Load outcomes and compute the dashboard.** Compute the outcome panels in your warehouse, since metrics backends cannot join to GitHub, from two tables: `agent_runs` (from `runs.jsonl`, with cost from telemetry grouped by session) and `pull_requests` (from the GitHub API, with the session parsed from the bundle and a 14-day follow-up flag as defined on [the metrics page](/en/org/metrics-frameworks/)).

    ```sql
    -- Weekly outcome panels per loop. A change is accepted when it merged and was
    -- not reverted or fixed within 14 days, so the last two weeks are provisional.
    WITH pr AS (  -- one row per session, so a run with several PRs is counted once
      SELECT repo, agent_session,
             count(*) AS prs,
             count(*) FILTER (WHERE merged_at IS NOT NULL
                                AND NOT followup_within_14d)    AS accepted
      FROM pull_requests
      WHERE agent_session IS NOT NULL
      GROUP BY repo, agent_session
    )
    SELECT
      date_trunc('week', r.started_at)                          AS week,
      r.loop_name,
      count(*)                                                  AS runs,
      avg(CASE WHEN r.completed_ok THEN 1.0 ELSE 0 END)         AS run_completion_rate,
      sum(CASE WHEN r.completed_ok AND p.prs > 0 THEN 1 ELSE 0 END)::numeric
        / NULLIF(sum(CASE WHEN r.completed_ok THEN 1 ELSE 0 END), 0) AS run_to_pr_rate,
      sum(coalesce(p.accepted, 0))                              AS accepted_changes,
      sum(r.cost_usd) / NULLIF(sum(coalesce(p.accepted, 0)), 0) AS run_cost_per_accepted_change
    FROM agent_runs r
    LEFT JOIN pr p
      ON p.repo = r.repo AND p.agent_session = r.session_id
    GROUP BY 1, 2
    ORDER BY 1 DESC, 2;
    ```

    `completed_ok` is `NOT is_error` for Claude Code and `NOT failed` for Codex. This query gives run cost only; the organization's full cost per accepted change adds seats and platform cost, per the [metrics definitions](/en/org/metrics-frameworks/) and [the economics page](/en/strategy/economics/).

6. **Wire incidents to runs.** When an incident is traced to a deploy, the chain is: deploy, merge commit, pull request, `provenance.session`, then every event with that `session.id` or `conversation.id`, ordered by `event.timestamp` (with `event.sequence` breaking ties) and grouped by `prompt.id` in Claude Code. The `tool_decision` events show whether a risky call was allowed by configuration, a hook, or a person. Add the session link to the incident record in your [agent incident process](/en/org/agent-incidents/), and tag the cause with the [failure taxonomy](/en/workflows/operate/failure-taxonomy/).

7. **Set retention and access per stream, then write it down.** See the template in the next section; configure it in the backends and in Claude Code's `cleanupPeriodDays` for local transcripts.

## What goes on the reference dashboard?

Eight panels answer a tech lead's weekly and a CTO's monthly questions. Group by loop, repository, and tool, never by person.

| Panel | Definition | Source | Watch for |
| --- | --- | --- | --- |
| Run completion rate | Runs that ended without error ÷ runs started | `runs.jsonl` | A drop after a model, prompt, or harness change |
| Run-to-PR rate | Runs that opened a pull request ÷ runs that completed | `runs.jsonl` + GitHub | Runs that finish but produce nothing reviewable |
| Accepted change rate | Merged and not followed up within 14 days ÷ pull requests opened | GitHub | The real throughput; the [definition](/en/org/metrics-frameworks/) is canonical |
| Run cost per accepted change | Run cost ÷ accepted changes (the SQL above) | Warehouse | Cost rising while acceptance stays flat |
| Spend by model and effort | `claude_code.cost.usage` by `model`, `effort`, `query_source`; `codex.turn.cost_microusd` by `reasoning_effort` | Metrics | Subagent or auxiliary spend growing unnoticed |
| Tool error rate | `tool_result` with `success=false` ÷ all, by `tool_name` | Events | A broken MCP server or test command burning turns |
| Permission friction | `tool_decision` rejects by `source`; `permission_mode_changed` events | Events | Loops blocked on approvals, or silent mode escalations |
| Budget and retry stops | `error_max_budget_usd` and `error_max_turns` subtypes; `claude_code.api_error` events | `runs.jsonl`, events | Budgets set too tight, or provider outages |

Set three alerts from day one: a single session above your cost ceiling, any `permission_mode_changed` event whose `to_mode` is `bypassPermissions` outside a sandboxed runner, and retry exhaustion above its weekly baseline. For a sense of scale, Anthropic's cost documentation (checked 2026-09-26) puts typical enterprise Claude Code spend at "around $13 per developer per active day and $150-250 per developer per month"; your own baseline after four weeks is the number to alert on.

## How long should you keep agent telemetry?

Retention follows what each stream contains. Treat this as a starting policy for your security and legal owners to adjust; obligations under [the EU AI Act](/en/org/eu-ai-act/) and sector rules in [regulated industries](/en/org/regulated-industries/) can require longer.

| Stream | Contains | Store | Starting retention | Who can read |
| --- | --- | --- | --- | --- |
| Metrics | Cost, tokens, sessions, lines, and pseudonymized users | Metrics backend | 13 months, for year-on-year comparison | Engineering |
| Operational events | `tool_result`, `api_request`, `api_error`, pseudonymized | Log store | 90 days | Engineering |
| Audit events | `tool_decision`, `permission_mode_changed`, `hook_execution_complete`, `auth`, `mcp_server_connection`, `plugin_installed`, `managed_settings_resolved`; `codex.tool_decision`, `codex.sandbox_outcome` | SIEM | Your security log policy (often one year or more) | Security |
| Content | Prompts, responses, tool output, and raw API bodies | Off by default; if on, a restricted store | Shortest workable, for example 30 days | Named incident responders |
| Change evidence | Evidence bundle, session link, and checkpoint trailers | Git and the code host | Life of the repository | Everyone with repository access |
| Local transcripts | Claude Code session files on developer machines | Disk | `cleanupPeriodDays` (default 30), pinned in managed settings | The developer |

Claude Code emits a `claude_code.retention_sweep` event each time it runs that cleanup; a nonzero `files_past_cutoff` means files outlived the configured period, which is itself an audit finding.

## Copy-paste prompts for agent observability

**Copy-paste prompt: reconstruct what an agent did before an incident**

```text
An incident was traced to merge commit $SHA on main. Reconstruct the agent run behind it.

1. Use `gh` to find the pull request that produced $SHA and print its evidence bundle.
   Read provenance.session, provenance.agent, provenance.model and provenance.task.
2. In ./telemetry/events.jsonl (an export of our collector's audit stream), select every
   record whose session.id or conversation.id equals that session. Sort by event.timestamp.
3. Produce a timeline table: time, prompt.id, event name, tool_name, decision and its
   source, success, and for Bash the command. Mark every tool call that touched files
   listed in the pull request's risk.touches.
4. Answer, citing record timestamps: which tool calls changed the code that failed,
   whether any permission was granted by a person rather than by config or a hook,
   and whether any check in the evidence bundle should have caught the failure.

Do not guess. If a record is missing, say which one and stop that line of reasoning.
Do not print email addresses; refer to people by their role in the bundle.
```

**Copy-paste prompt: the weekly telemetry triage**

```text
Read ./reports/agent-dashboard-week.csv (columns: week, loop_name, runs,
run_completion_rate, run_to_pr_rate, accepted_changes, run_cost_per_accepted_change,
tool_error_rate, budget_stops) covering the last eight weeks.

For each loop:
- Flag any metric that moved more than 20% against its own four-week median.
  Ignore the last two weeks for accepted_changes and cost per accepted change;
  they are provisional because of the 14-day follow-up window.
- For each flag, name the most likely harness cause (prompt, tool permissions,
  MCP server, test command, model or effort change, budget) and the one query
  against ./telemetry/events.jsonl that would confirm or rule it out. Run the query.
- Recommend one action per flagged loop: keep, fix the harness, or lower its autonomy
  level, with the evidence for it.

Output a table plus at most five sentences. Never rank or name individual developers.
```

**Copy-paste prompt: audit a repository's agent instrumentation**

```text
Audit this repository's agent instrumentation against our telemetry standard.
Read every workflow in .github/workflows/ that runs claude, codex, or entire, the
evidence bundle template in .github/pull_request_template.md, and .claude/settings.json.

Report, as a checklist with file and line for each finding:
1. Every agent run captures a session ID (claude --session-id or codex exec --json)
   and writes it into the evidence bundle's provenance.session.
2. Every run appends a record to runs.jsonl with status, turns and cost or tokens.
3. OTEL_RESOURCE_ATTRIBUTES sets ci.run_id and loop.name, and
   OTEL_METRICS_INCLUDE_RESOURCE_ATTRIBUTES is false in CI.
4. No OTLP header, API key, or collector token appears in a committed file.
5. No OTEL_* exporter variable is set in .claude/settings.json (Claude Code ignores
   them there, so it is a sign someone expected it to work).
6. Content logging (OTEL_LOG_USER_PROMPTS, OTEL_LOG_RAW_API_BODIES, and Codex
   log_user_prompt) is off unless the workflow has a comment citing an approval.

Propose the smallest diff that fixes each failed item. Do not change the prompts.
```

## How do you know the telemetry is right?

A dashboard that reads zero or double is worse than none. Verify the pipeline itself:

- **Arrival.** After any configuration change, check for `claude_code.session.count` (metrics) or `claude_code.user_prompt` (events). If nothing arrives, start Claude Code with `claude --debug-file /tmp/claude-otel.log` and look for `[3P telemetry]` errors. For Codex, run one `codex exec` and look for `codex.conversation_starts`.
- **Join coverage.** Each week, count merged agent pull requests whose `provenance.session` resolves to at least one telemetry event. Below 95%, a workflow is dropping its session ID; the instrumentation audit prompt finds it.
- **Cost reconciliation.** Claude Code's docs call cost metrics approximations. Once a month, compare summed `claude_code.cost.usage` and Codex's estimated cost with the invoice or the Console, per workspace. A gap above a few percent usually means unreported CI runs or a second billing path.
- **Canary run.** A scheduled CI job runs a fixed, cheap prompt once a day and fails if its `runs.jsonl` record and its session's events are not in the warehouse within an hour.

| Who | Owns | Signs |
| --- | --- | --- |
| Platform or developer-experience team | Collector, managed settings, `config.toml` templates, and the canary | Pipeline changes |
| Tech lead | Loop names, the dashboard review in the weekly retro | Autonomy changes based on the panels |
| Security | Audit pipeline, SIEM rules, and content-logging approvals | Retention and access |
| CTO | The measurement charter and the monthly cost view | Budget ceilings and the charter |

## What breaks when you instrument coding agents?

**Telemetry is enabled but nothing arrives.** Claude Code has no default OTLP protocol, so every `otlp` exporter needs `OTEL_EXPORTER_OTLP_PROTOCOL` or its per-signal variant. Recovery: set the protocol, then check the debug file for `[3P telemetry]` errors. Lines prefixed `[Anthropic telemetry]` are Anthropic's own operational telemetry, not your pipeline.

**Metric storage costs explode.** Per-run IDs in `OTEL_RESOURCE_ATTRIBUTES` and the default `session.id` on metrics create a new series per run. Recovery: set `OTEL_METRICS_INCLUDE_RESOURCE_ATTRIBUTES=false` and, in large organizations, `OTEL_METRICS_INCLUDE_SESSION_ID=false`; keep run and session IDs on events, where cardinality is cheap.

**A secret shows up in the log store.** `OTEL_LOG_TOOL_DETAILS=1` records full Bash commands, and a token passed on a command line goes with it. Recovery: rotate the secret, purge the records, route tool details only to the audit pipeline, and add a collector rule or SIEM alert for credential patterns. Never enable `OTEL_LOG_RAW_API_BODIES` outside a time-boxed investigation; it exports the whole conversation.

**The cost panel disagrees with the invoice.** Telemetry cost is a client-side estimate, Claude Code before v2.1.214 double-counted cost and tokens on streams that emit multiple cumulative `message_delta` frames, and CI runs without telemetry are invisible. Recovery: upgrade, reconcile monthly, and report the invoice as the authoritative number with telemetry as the breakdown.

**Commits do not join to sessions.** Squash and rebase merges rewrite SHAs, and Codex CLI 0.157.1 emits no commit SHA in its telemetry. Recovery: join on `provenance.session` in the evidence bundle, and fail the pull request check when the field is empty.

**The dashboard becomes a leaderboard.** Someone sorts cost by developer. Recovery: the charter forbids it, the analytics pipeline carries only a keyed hash of the email with the account and install IDs deleted, and the per-user view exists only in the SIEM for security investigations.

**Codex metrics go somewhere you did not choose.** Leaving `metrics_exporter` unset in 0.157.1 means the `statsig` default. Recovery: set it explicitly to your collector or to `"none"` in the `config.toml` template you distribute.

## Where to go next with agent observability

Telemetry tells you what the runs did. The next pages use it to classify failures, handle incidents, and keep the codebase healthy.

- [A failure taxonomy for agent-written changes](/en/workflows/operate/failure-taxonomy/) — Tag each incident with the failure class and the earliest control that should have caught it.
- [When an agent causes an incident](/en/org/agent-incidents/) — Containment, attribution through provenance, and a postmortem that names the failed control.
- [The evidence bundle](/en/workflows/ship/evidence-bundle/) — The pull request contract whose provenance fields carry the session join key.
- [Metrics frameworks](/en/org/metrics-frameworks/) — Canonical definitions of accepted change rate and cost per accepted change.
- [Keeping an agent-written codebase healthy](/en/workflows/operate/codebase-health/) — The trend signals that sit beside these run-level panels.
- [Agent telemetry tools](/en/ecosystem/observability/agent-telemetry/) — The ecosystem shelf: trace platforms, usage trackers, and gateways that consume this data.
