Observing the agents: traces, tool calls, cost and audit logs

Agent observability means exporting each coding agent’s telemetry (sessions, model requests, tool calls, permission decisions, and cost) through OpenTelemetry, stamping every run with IDs that join it to its pull request and to later incidents, and keeping audit events longer than analytics. Claude Code and Codex export OpenTelemetry natively; for Cursor, hooks and commit provenance fill the gap.

This page is for the developer who runs agent loops in CI, the tech lead who owns the loop’s autonomy, and the CTO who signs the budget and the audit policy. The situation it solves: an agent-written change caused a production incident on Tuesday, and on Wednesday nobody can say which session wrote it, what the agent ran, who approved the tool calls, or what that loop costs per change that survives. You have logs for your services and none for the things writing them.

What you get from agent observability done this way

• A collector with a pseudonymized analytics pipeline and an identified audit pipeline.
• Export settings for Claude Code, Codex, and the verified path for Cursor.
• Four join keys that lead from a commit to its pull request, session, and tool calls.
• An eight-panel dashboard with its SQL, a retention template, and three copy-paste prompts.

Why do agent runs need their own telemetry?

Your service telemetry answers “what did the system do?”. Agent telemetry answers a different question: “what did the thing that changed the system do, on whose authority, and at what cost?”. Faros AI’s AI Engineering Report 2026: The Acceleration Whiplash (April 2026, two years of telemetry from 22,000 developers on Faros’s own platform) reports median time in review up 441.5% and incidents per pull request up 242.7%. When reviewers cannot read everything, evidence about the run has to come from telemetry.

Three things make agent telemetry different:

• The unit is a run, not a request. One run is a session of many model calls and tool calls, sometimes with subagents. Success is decided days later, when the pull request merges and survives.
• Identity is a developer’s. Claude Code attributes every tool call to the developer who started the session; it does not act under a separate service account (Claude Code monitoring docs, checked 2026-09-26). The same data is both an audit trail and a potential surveillance tool.
• Content is sensitive by default. Prompts, tool arguments, and model responses can contain secrets and customer data. Both tools keep content out of telemetry unless you opt in.

What does each tool emit?

Configure each tool separately and normalize afterwards.

Concept	Claude Code 2.1.283	Codex CLI 0.157.1	Cursor
Export mechanism	OTel metrics and log events; traces in beta	OTel logs, traces, and metrics from the [otel] table	No OTel export verified on 2026-09-26; use hooks and provenance tools
Session or run ID	session.id (OTel); session_id in claude -p --output-format json	conversation.id (OTel); thread_id in codex exec --json	Cloud Agents API run IDs and statusChange webhooks (verified 2026-08-28)
Cost	claude_code.cost.usage (USD); total_cost_usd per headless run	codex.turn.cost_microusd metric; codex.turn_cost event with usage.estimated_usd	Not verified
Tokens	claude_code.token.usage by type, model, effort	codex.turn.token_usage; usage on each turn.completed	Not verified
Tool calls	claude_code.tool_result (tool_name, success, duration_ms)	codex.tool_result; codex.tool.call metric	Hooks (JSON over stdio); event names not verified
Permission decisions	claude_code.tool_decision, claude_code.permission_mode_changed	codex.tool_decision, codex.sandbox_outcome	Not verified
Link to a commit	vcs.ref.head.revision on the tool_result of a successful git commit (needs OTEL_LOG_TOOL_DETAILS=1)	None; record the thread_id in the pull request	Entire-Checkpoint commit trailer from Entire CLI
Vendor dashboard	claude.ai/analytics/claude-code (Team, Enterprise); platform.claude.com/claude-code (Console)	Enterprise analytics dashboard (secondary: OpenAI help pages seen only in search extracts)	Not verified on 2026-09-26 (cursor.com unreachable)

Sources: Claude Code monitoring and analytics docs; Codex source at tag rust-v0.157.1 (codex-rs/otel, codex-rs/config/src/types.rs, codex-rs/exec/src/exec_events.rs), all read 2026-09-26. cursor.com was unreachable on 2026-09-26, so Cursor’s analytics and audit endpoints are not named here.

Where the OpenTelemetry GenAI conventions stand

The shared vocabulary for agent spans (invoke_agent, execute_tool, plan) and for MCP (mcp.method.name, gen_ai.tool.name) lives in the open-telemetry/semantic-conventions-genai repository, which had no release on 2026-09-26; both documents carry Status: Development. Claude Code adds a few GenAI attributes, such as gen_ai.request.model, as aliases on its spans; Codex uses codex.* names. Build your dashboard on a small normalized view (step 5) rather than on either vendor’s names, and remap when the conventions stabilize.

Set up agent observability, step by step

1. Write the measurement charter before you collect anything. One paragraph, signed by the CTO: which questions the data answers (run success, cost per accepted change, audit), who sees identities (the security team, through the SIEM), who sees pseudonyms (everyone else), and that no metric is used to rate an individual. The metrics frameworks page explains why PR counts and token counts become gamed targets, and privacy and data handling covers vendor data terms and retention. Telemetry that engineers believe is a surveillance tool gets switched off in their shells.

2. Stand up one OpenTelemetry Collector with split pipelines. Every tool sends OTLP to the same collector. For the analytics backends the collector replaces the email with a keyed hash, deletes the other user identifiers, and drops tool arguments; the SIEM gets the unmodified audit stream:

   # otel-collector.yaml (OpenTelemetry Collector contrib distribution)
   receivers:
     otlp:
       protocols:
         grpc: { endpoint: 0.0.0.0:4317 }
         http: { endpoint: 0.0.0.0:4318 }
   
   processors:
     batch: {}
     # Keyed hash: PSEUDONYM_SALT is a secret only the collector holds.
     transform/pseudonymize:
       error_mode: ignore
       log_statements:
         - context: log
           statements:
             - set(attributes["user.email"], SHA256(Concat([attributes["user.email"], "${env:PSEUDONYM_SALT}"], ""))) where attributes["user.email"] != nil
       metric_statements:
         - context: datapoint
           statements:
             - set(attributes["user.email"], SHA256(Concat([attributes["user.email"], "${env:PSEUDONYM_SALT}"], ""))) where attributes["user.email"] != nil
     attributes/drop-ids:
       actions:
         - key: user.account_uuid
           action: delete
         - key: user.account_id
           action: delete
         - key: user.id
           action: delete
         - key: enduser.id
           action: delete
     attributes/strip-content:
       actions:
         - key: tool_parameters
           action: delete
         - key: tool_input
           action: delete
         - key: error
           action: delete
   
   exporters:
     prometheus:
       endpoint: 0.0.0.0:8889
     otlphttp/analytics:
       endpoint: https://logs.internal.example.com/otlp
     otlphttp/siem:
       endpoint: https://siem.internal.example.com:4318
   
   service:
     pipelines:
       metrics:
         receivers: [otlp]
         processors: [transform/pseudonymize, attributes/drop-ids, batch]
         exporters: [prometheus]
       logs/analytics:
         receivers: [otlp]
         processors: [transform/pseudonymize, attributes/drop-ids, attributes/strip-content, batch]
         exporters: [otlphttp/analytics]
       logs/audit:
         receivers: [otlp]
         processors: [batch]
         exporters: [otlphttp/siem]

   Claude Code attaches user.email, user.account_uuid, user.account_id (the admin-API ID), and user.id (a persistent per-install ID) to every metric and event; hashing only the email leaves three ways back to each person. The attributes processor’s hash action is unsalted, so a list of staff emails reverses it. logs.internal.example.com and siem.internal.example.com stand for your log store and your SIEM’s OTLP receiver.

3. Turn on export in each tool. Developer machines get it through managed configuration; CI jobs set it in the job.

   Claude Code

   Put the exporter settings in managed settings. Claude Code ignores OpenTelemetry exporter variables in a repository’s .claude/settings.json and .claude/settings.local.json, so a repository cannot turn telemetry on, redirect it, or capture content. When managed settings set OTEL_EXPORTER_OTLP_ENDPOINT, Claude Code removes developer-set per-signal endpoints at startup (v2.1.217 and later).

   {
     "env": {
       "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
       "OTEL_METRICS_EXPORTER": "otlp",
       "OTEL_LOGS_EXPORTER": "otlp",
       "OTEL_EXPORTER_OTLP_PROTOCOL": "grpc",
       "OTEL_EXPORTER_OTLP_ENDPOINT": "https://otel-collector.internal.example.com:4317",
       "OTEL_METRICS_INCLUDE_REPOSITORY": "true",
       "OTEL_LOG_TOOL_DETAILS": "1"
     }
   }

   OTEL_LOG_TOOL_DETAILS=1 adds Bash commands, MCP server and tool names, and tool input to events. It is what makes MCP activity auditable and what puts the commit SHA on git commit tool results; it is also where secrets typed on a command line end up, which is why the collector in step 2 strips those attributes from everything except the SIEM stream, and why the endpoint must be TLS (https://) or reachable only over a private network. Prompt text stays off unless you set OTEL_LOG_USER_PROMPTS=1.

   For a headless CI run, set the same variables on the job, and give the run its own session ID so the pull request can carry it:

   .github/workflows/agent-issue-to-pr.yml

   name: agent-issue-to-pr
   on:
     workflow_dispatch:
       inputs:
         issue:
           description: Issue number to implement
           required: true
           type: number
   permissions: {}
   env:
     ISSUE: ${{ inputs.issue }}
     BRANCH: agent/issue-${{ inputs.issue }}
   jobs:
     issue-to-pr:
       # Runs the agent. Read-only token: nothing here can push.
       runs-on: ubuntu-latest
       timeout-minutes: 30
       permissions:
         contents: read
         issues: read
       outputs:
         session_id: ${{ steps.sid.outputs.id }}
       env:
         CLAUDE_CODE_ENABLE_TELEMETRY: "1"
         OTEL_METRICS_EXPORTER: otlp
         OTEL_LOGS_EXPORTER: otlp
         OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
         OTEL_EXPORTER_OTLP_ENDPOINT: ${{ secrets.OTEL_ENDPOINT }}
         OTEL_EXPORTER_OTLP_HEADERS: ${{ secrets.OTEL_HEADERS }}
         OTEL_RESOURCE_ATTRIBUTES: ci.run_id=${{ github.run_id }},loop.name=issue-to-pr
         OTEL_METRICS_INCLUDE_RESOURCE_ATTRIBUTES: "false"
       steps:
         - uses: actions/checkout@v7
           with:
             persist-credentials: false
         - uses: actions/setup-node@v7
           with:
             node-version: 22
         - name: Install Claude Code
           run: npm install -g @anthropic-ai/claude-code
         - name: Prepare the branch and the prompt
           env:
             GH_TOKEN: ${{ github.token }}
           run: |
             git config user.name "issue-to-pr agent"
             git config user.email "issue-to-pr-agent@users.noreply.github.com"
             git switch -c "$BRANCH"
             { cat .github/prompts/issue-to-pr.md; echo
               gh issue view "$ISSUE" --json number,title,body \
                 --jq '"Issue #\(.number): \(.title)\n\n\(.body)"'
             } > "$RUNNER_TEMP/prompt.md"
         - name: Pick a session ID
           id: sid
           run: echo "id=$(cat /proc/sys/kernel/random/uuid)" >> "$GITHUB_OUTPUT"
         - name: Run the agent
           env:
             ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
             SID: ${{ steps.sid.outputs.id }}
           run: |
             status=0
             claude -p "$(cat "$RUNNER_TEMP/prompt.md")" \
               --session-id "$SID" --output-format json \
               --max-budget-usd 10 --permission-mode dontAsk \
               --allowedTools "Read,Grep,Glob,Edit,Write,Bash(npm test *),Bash(git commit *)" \
               > "$RUNNER_TEMP/run.json" || status=$?
             jq -c '{session_id, is_error, subtype, num_turns, total_cost_usd, duration_ms}' \
               "$RUNNER_TEMP/run.json" >> "$RUNNER_TEMP/runs.jsonl"
             exit "$status"
         - name: Keep the run record
           if: always()
           uses: actions/upload-artifact@v7
           with:
             name: agent-run-${{ github.run_id }}
             path: ${{ runner.temp }}/runs.jsonl
             if-no-files-found: ignore
         - name: Bundle the agent's commits
           env:
             BASE: ${{ github.sha }}
           run: git bundle create "$RUNNER_TEMP/agent.bundle" "$BRANCH" "^$BASE"
         - uses: actions/upload-artifact@v7
           with:
             name: agent-commits-${{ github.run_id }}
             path: ${{ runner.temp }}/agent.bundle
             if-no-files-found: error
   
     open-pr:
       # Holds the write token. Runs no code from the agent's commits.
       needs: issue-to-pr
       runs-on: ubuntu-latest
       timeout-minutes: 5
       permissions:
         contents: write
         pull-requests: write
       steps:
         - uses: actions/checkout@v7
           with:
             persist-credentials: false
         - uses: actions/download-artifact@v7
           with:
             name: agent-commits-${{ github.run_id }}
             path: ${{ runner.temp }}
         - name: Push and open the pull request
           env:
             GH_TOKEN: ${{ github.token }}
             AGENT_SESSION_ID: ${{ needs.issue-to-pr.outputs.session_id }}
           run: |
             git fetch "$RUNNER_TEMP/agent.bundle" "$BRANCH:$BRANCH"
             gh auth setup-git
             git push origin "refs/heads/$BRANCH"
             gh pr create --head "$BRANCH" --title "Fix #$ISSUE (agent)" \
               --body "Closes #$ISSUE. Agent session: $AGENT_SESSION_ID"

   OTEL_METRICS_INCLUDE_RESOURCE_ATTRIBUTES=false keeps ci.run_id on events but off metric labels, where it would create one time series per CI run. The subtype field tells you why a run stopped: success, error_max_turns, error_max_budget_usd, error_during_execution, or error_max_structured_output_retries. The run is started by hand (workflow_dispatch needs write access to the repository) and checks out the ref chosen at dispatch, which is the default branch unless the person starting it picks another in “Use workflow from” or passes gh workflow run --ref. Steps inside the agent job are not isolated from each other: Edit, Write and Bash(npm test *) let the agent rewrite the test script and run any code it likes, the issue text it works from is untrusted, and that code can read ANTHROPIC_API_KEY and the OTLP headers and write to $GITHUB_ENV or $GITHUB_PATH, which every later step of the same job inherits. That is why the agent job holds only a read-only GITHUB_TOKEN, and the push lives in a separate open-pr job: it receives the commits as a git bundle artifact, gets the session ID from a step that ran before the agent, and runs no code from those commits. If the agent made no commits, git bundle create refuses an empty bundle and the run stops before open-pr. Read the issue before you dispatch the run; for workflows triggered by untrusted contributors, follow the CI security rules first. gh pr create with GITHUB_TOKEN works only when “Allow GitHub Actions to create and approve pull requests” is enabled in the repository’s or organization’s Actions settings, and a pull request opened with that token does not trigger pull_request workflows, so the agent’s PR gets no CI run. If CI must run on it, open the PR with a GitHub App token or a fine-grained personal access token instead. The run record is written even when the agent stops with an error, so failed runs reach runs.jsonl too.

   Traces are beta: add CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 and OTEL_TRACES_EXPORTER=otlp. In claude -p and Agent SDK sessions, Claude Code reads TRACEPARENT from its environment, so its claude_code.interaction spans nest under your CI job’s trace.

   Codex

   Codex reads an [otel] table from config.toml. In 0.157.1 the log and trace exporters default to none, and metrics_exporter defaults to statsig, a Codex-internal metrics destination; set all three explicitly so metrics go where your policy says.

   # ~/.codex/config.toml (or written by the CI job at run time)
   [otel]
   environment = "ci"
   log_user_prompt = false
   exporter = { otlp-http = { endpoint = "https://otel-collector.internal.example.com:4318/v1/logs", protocol = "binary" } }
   trace_exporter = { otlp-http = { endpoint = "https://otel-collector.internal.example.com:4318/v1/traces", protocol = "binary" } }
   metrics_exporter = { otlp-http = { endpoint = "https://otel-collector.internal.example.com:4318/v1/metrics", protocol = "binary" } }
   
   [otel.span_attributes]
   "loop.name" = "issue-to-pr"

   Collector auth headers go in plain text in an inline headers = { ... } table per exporter, so write the file in the CI job from a secret and never commit it. Codex’s main branch adds log_agent_responses and log_guardian_assessments opt-ins after 0.157.1; they are not in 0.157.1 (checked 2026-09-26), so do not copy them into a config for that release.

   For headless runs, codex exec --json prints JSONL events. The first is thread.started with the thread_id, which is the same thread ID the OTel events carry as conversation.id. turn.completed carries token usage and turn.failed or error marks a failed run:

   # CI step (Codex CLI 0.157.1), with OPENAI_API_KEY from a secret
   codex exec --json --sandbox workspace-write \
     "$(cat .github/prompts/issue-to-pr.md)" > events.jsonl
   jq -c -s '{
     thread_id: (map(select(.type == "thread.started"))[0].thread_id),
     failed: any(.[]; .type == "turn.failed" or .type == "error"),
     output_tokens: (map(select(.type == "turn.completed") | .usage.output_tokens) | add)
   }' events.jsonl >> runs.jsonl

   Run one smoke test per CI image and confirm the codex.* events reach the collector from codex exec, not only from interactive sessions, before you build panels on them (an open report, openai/codex#12913, says codex exec emits no OTel metrics; status unverified on 2026-09-26). If the metrics are absent, build Codex cost panels on the output_tokens usage recorded in runs.jsonl instead.

   Cursor

   On 2026-09-26 this page could verify no OpenTelemetry export for Cursor, so the verified path is provenance plus the cloud API:

   • Hooks. Cursor runs hooks that receive agent events as JSON over stdio (verified 2026-08-28). A hook can append each event to a log file your collector tails.

   • Entire CLI (entireio/cli, v0.11.3, 5.1k GitHub stars on 2026-09-26) installs git hooks plus agent hooks in .cursor/hooks.json. When the agent commits, it links prompts, transcript, tool calls, files touched, and tokens to the commit through an Entire-Checkpoint: trailer:

     # Terminal, in the repository (install first: brew install --cask entireio/tap/entire)
     entire enable --agent cursor
     entire status

     Entire stores transcripts in your git repository and redacts secrets only partially. On a public repository, point it at a private one with --checkpoint-remote.

   • Cloud Agents. Runs started through the Cloud Agents API report statusChange webhooks (verified 2026-08-28). Record the run ID and final status in runs.jsonl exactly as the other tabs do.

4. Stamp every run with four join keys, and write them into the pull request. Telemetry you cannot join to outcomes only tells you what you spent. Each run carries:

   Key	Claude Code	Codex	Cursor	Where it lands
   Session	--session-id UUID	thread_id	Entire checkpoint ID or cloud run ID	Evidence bundle provenance.session
   CI run	OTEL_RESOURCE_ATTRIBUTES ci.run_id	[otel.span_attributes]	runs.jsonl	runs.jsonl, events
   Loop	loop.name resource attribute	loop.name span attribute	runs.jsonl	Dashboard grouping
   Task	Issue URL in the prompt and the bundle	Same	Same	Evidence bundle provenance.task

   The pull request’s evidence bundle already has a provenance.session field; the CI job fills it from AGENT_SESSION_ID or the thread_id. Join on the session, not on the commit SHA: a squash merge creates a new SHA, so the SHA recorded at git commit time never appears on the default branch. On Team and Enterprise plans with the GitHub app, Claude Code’s analytics also labels merged pull requests claude-code-assisted, a cross-check that is unavailable under Zero Data Retention.

5. Load outcomes and compute the dashboard. Compute the outcome panels in your warehouse, since metrics backends cannot join to GitHub, from two tables: agent_runs (from runs.jsonl, with cost from telemetry grouped by session) and pull_requests (from the GitHub API, with the session parsed from the bundle and a 14-day follow-up flag as defined on the metrics page).

   -- Weekly outcome panels per loop. A change is accepted when it merged and was
   -- not reverted or fixed within 14 days, so the last two weeks are provisional.
   WITH pr AS (  -- one row per session, so a run with several PRs is counted once
     SELECT repo, agent_session,
            count(*) AS prs,
            count(*) FILTER (WHERE merged_at IS NOT NULL
                               AND NOT followup_within_14d)    AS accepted
     FROM pull_requests
     WHERE agent_session IS NOT NULL
     GROUP BY repo, agent_session
   )
   SELECT
     date_trunc('week', r.started_at)                          AS week,
     r.loop_name,
     count(*)                                                  AS runs,
     avg(CASE WHEN r.completed_ok THEN 1.0 ELSE 0 END)         AS run_completion_rate,
     sum(CASE WHEN r.completed_ok AND p.prs > 0 THEN 1 ELSE 0 END)::numeric
       / NULLIF(sum(CASE WHEN r.completed_ok THEN 1 ELSE 0 END), 0) AS run_to_pr_rate,
     sum(coalesce(p.accepted, 0))                              AS accepted_changes,
     sum(r.cost_usd) / NULLIF(sum(coalesce(p.accepted, 0)), 0) AS run_cost_per_accepted_change
   FROM agent_runs r
   LEFT JOIN pr p
     ON p.repo = r.repo AND p.agent_session = r.session_id
   GROUP BY 1, 2
   ORDER BY 1 DESC, 2;

   completed_ok is NOT is_error for Claude Code and NOT failed for Codex. This query gives run cost only; the organization’s full cost per accepted change adds seats and platform cost, per the metrics definitions and the economics page.

6. Wire incidents to runs. When an incident is traced to a deploy, the chain is: deploy, merge commit, pull request, provenance.session, then every event with that session.id or conversation.id, ordered by event.timestamp (with event.sequence breaking ties) and grouped by prompt.id in Claude Code. The tool_decision events show whether a risky call was allowed by configuration, a hook, or a person. Add the session link to the incident record in your agent incident process, and tag the cause with the failure taxonomy.

7. Set retention and access per stream, then write it down. See the template in the next section; configure it in the backends and in Claude Code’s cleanupPeriodDays for local transcripts.

What goes on the reference dashboard?

Eight panels answer a tech lead’s weekly and a CTO’s monthly questions. Group by loop, repository, and tool, never by person.

Panel	Definition	Source	Watch for
Run completion rate	Runs that ended without error ÷ runs started	runs.jsonl	A drop after a model, prompt, or harness change
Run-to-PR rate	Runs that opened a pull request ÷ runs that completed	runs.jsonl + GitHub	Runs that finish but produce nothing reviewable
Accepted change rate	Merged and not followed up within 14 days ÷ pull requests opened	GitHub	The real throughput; the definition is canonical
Run cost per accepted change	Run cost ÷ accepted changes (the SQL above)	Warehouse	Cost rising while acceptance stays flat
Spend by model and effort	claude_code.cost.usage by model, effort, query_source; codex.turn.cost_microusd by reasoning_effort	Metrics	Subagent or auxiliary spend growing unnoticed
Tool error rate	tool_result with success=false ÷ all, by tool_name	Events	A broken MCP server or test command burning turns
Permission friction	tool_decision rejects by source; permission_mode_changed events	Events	Loops blocked on approvals, or silent mode escalations
Budget and retry stops	error_max_budget_usd and error_max_turns subtypes; claude_code.api_error events	runs.jsonl, events	Budgets set too tight, or provider outages

Set three alerts from day one: a single session above your cost ceiling, any permission_mode_changed event whose to_mode is bypassPermissions outside a sandboxed runner, and retry exhaustion above its weekly baseline. For a sense of scale, Anthropic’s cost documentation (checked 2026-09-26) puts typical enterprise Claude Code spend at “around $13 per developer per active day and $150-250 per developer per month”; your own baseline after four weeks is the number to alert on.

How long should you keep agent telemetry?

Retention follows what each stream contains. Treat this as a starting policy for your security and legal owners to adjust; obligations under the EU AI Act and sector rules in regulated industries can require longer.

Stream	Contains	Store	Starting retention	Who can read
Metrics	Cost, tokens, sessions, lines, and pseudonymized users	Metrics backend	13 months, for year-on-year comparison	Engineering
Operational events	tool_result, api_request, api_error, pseudonymized	Log store	90 days	Engineering
Audit events	tool_decision, permission_mode_changed, hook_execution_complete, auth, mcp_server_connection, plugin_installed, managed_settings_resolved; codex.tool_decision, codex.sandbox_outcome	SIEM	Your security log policy (often one year or more)	Security
Content	Prompts, responses, tool output, and raw API bodies	Off by default; if on, a restricted store	Shortest workable, for example 30 days	Named incident responders
Change evidence	Evidence bundle, session link, and checkpoint trailers	Git and the code host	Life of the repository	Everyone with repository access
Local transcripts	Claude Code session files on developer machines	Disk	cleanupPeriodDays (default 30), pinned in managed settings	The developer

Claude Code emits a claude_code.retention_sweep event each time it runs that cleanup; a nonzero files_past_cutoff means files outlived the configured period, which is itself an audit finding.

Copy-paste prompts for agent observability

Copy-paste prompt: reconstruct what an agent did before an incident

An incident was traced to merge commit $SHA on main. Reconstruct the agent run behind it.

1. Use `gh` to find the pull request that produced $SHA and print its evidence bundle.
   Read provenance.session, provenance.agent, provenance.model and provenance.task.
2. In ./telemetry/events.jsonl (an export of our collector's audit stream), select every
   record whose session.id or conversation.id equals that session. Sort by event.timestamp.
3. Produce a timeline table: time, prompt.id, event name, tool_name, decision and its
   source, success, and for Bash the command. Mark every tool call that touched files
   listed in the pull request's risk.touches.
4. Answer, citing record timestamps: which tool calls changed the code that failed,
   whether any permission was granted by a person rather than by config or a hook,
   and whether any check in the evidence bundle should have caught the failure.

Do not guess. If a record is missing, say which one and stop that line of reasoning.
Do not print email addresses; refer to people by their role in the bundle.

Copy-paste prompt: the weekly telemetry triage

Read ./reports/agent-dashboard-week.csv (columns: week, loop_name, runs,
run_completion_rate, run_to_pr_rate, accepted_changes, run_cost_per_accepted_change,
tool_error_rate, budget_stops) covering the last eight weeks.

For each loop:
- Flag any metric that moved more than 20% against its own four-week median.
  Ignore the last two weeks for accepted_changes and cost per accepted change;
  they are provisional because of the 14-day follow-up window.
- For each flag, name the most likely harness cause (prompt, tool permissions,
  MCP server, test command, model or effort change, budget) and the one query
  against ./telemetry/events.jsonl that would confirm or rule it out. Run the query.
- Recommend one action per flagged loop: keep, fix the harness, or lower its autonomy
  level, with the evidence for it.

Output a table plus at most five sentences. Never rank or name individual developers.

Copy-paste prompt: audit a repository's agent instrumentation

Audit this repository's agent instrumentation against our telemetry standard.
Read every workflow in .github/workflows/ that runs claude, codex, or entire, the
evidence bundle template in .github/pull_request_template.md, and .claude/settings.json.

Report, as a checklist with file and line for each finding:
1. Every agent run captures a session ID (claude --session-id or codex exec --json)
   and writes it into the evidence bundle's provenance.session.
2. Every run appends a record to runs.jsonl with status, turns and cost or tokens.
3. OTEL_RESOURCE_ATTRIBUTES sets ci.run_id and loop.name, and
   OTEL_METRICS_INCLUDE_RESOURCE_ATTRIBUTES is false in CI.
4. No OTLP header, API key, or collector token appears in a committed file.
5. No OTEL_* exporter variable is set in .claude/settings.json (Claude Code ignores
   them there, so it is a sign someone expected it to work).
6. Content logging (OTEL_LOG_USER_PROMPTS, OTEL_LOG_RAW_API_BODIES, and Codex
   log_user_prompt) is off unless the workflow has a comment citing an approval.

Propose the smallest diff that fixes each failed item. Do not change the prompts.

How do you know the telemetry is right?

A dashboard that reads zero or double is worse than none. Verify the pipeline itself:

• Arrival. After any configuration change, check for claude_code.session.count (metrics) or claude_code.user_prompt (events). If nothing arrives, start Claude Code with claude --debug-file /tmp/claude-otel.log and look for [3P telemetry] errors. For Codex, run one codex exec and look for codex.conversation_starts.
• Join coverage. Each week, count merged agent pull requests whose provenance.session resolves to at least one telemetry event. Below 95%, a workflow is dropping its session ID; the instrumentation audit prompt finds it.
• Cost reconciliation. Claude Code’s docs call cost metrics approximations. Once a month, compare summed claude_code.cost.usage and Codex’s estimated cost with the invoice or the Console, per workspace. A gap above a few percent usually means unreported CI runs or a second billing path.
• Canary run. A scheduled CI job runs a fixed, cheap prompt once a day and fails if its runs.jsonl record and its session’s events are not in the warehouse within an hour.

Who	Owns	Signs
Platform or developer-experience team	Collector, managed settings, config.toml templates, and the canary	Pipeline changes
Tech lead	Loop names, the dashboard review in the weekly retro	Autonomy changes based on the panels
Security	Audit pipeline, SIEM rules, and content-logging approvals	Retention and access
CTO	The measurement charter and the monthly cost view	Budget ceilings and the charter

What breaks when you instrument coding agents?

Telemetry is enabled but nothing arrives. Claude Code has no default OTLP protocol, so every otlp exporter needs OTEL_EXPORTER_OTLP_PROTOCOL or its per-signal variant. Recovery: set the protocol, then check the debug file for [3P telemetry] errors. Lines prefixed [Anthropic telemetry] are Anthropic’s own operational telemetry, not your pipeline.

Metric storage costs explode. Per-run IDs in OTEL_RESOURCE_ATTRIBUTES and the default session.id on metrics create a new series per run. Recovery: set OTEL_METRICS_INCLUDE_RESOURCE_ATTRIBUTES=false and, in large organizations, OTEL_METRICS_INCLUDE_SESSION_ID=false; keep run and session IDs on events, where cardinality is cheap.

A secret shows up in the log store. OTEL_LOG_TOOL_DETAILS=1 records full Bash commands, and a token passed on a command line goes with it. Recovery: rotate the secret, purge the records, route tool details only to the audit pipeline, and add a collector rule or SIEM alert for credential patterns. Never enable OTEL_LOG_RAW_API_BODIES outside a time-boxed investigation; it exports the whole conversation.

The cost panel disagrees with the invoice. Telemetry cost is a client-side estimate, Claude Code before v2.1.214 double-counted cost and tokens on streams that emit multiple cumulative message_delta frames, and CI runs without telemetry are invisible. Recovery: upgrade, reconcile monthly, and report the invoice as the authoritative number with telemetry as the breakdown.

Commits do not join to sessions. Squash and rebase merges rewrite SHAs, and Codex CLI 0.157.1 emits no commit SHA in its telemetry. Recovery: join on provenance.session in the evidence bundle, and fail the pull request check when the field is empty.

The dashboard becomes a leaderboard. Someone sorts cost by developer. Recovery: the charter forbids it, the analytics pipeline carries only a keyed hash of the email with the account and install IDs deleted, and the per-user view exists only in the SIEM for security investigations.

Codex metrics go somewhere you did not choose. Leaving metrics_exporter unset in 0.157.1 means the statsig default. Recovery: set it explicitly to your collector or to "none" in the config.toml template you distribute.

Where to go next with agent observability

Telemetry tells you what the runs did. The next pages use it to classify failures, handle incidents, and keep the codebase healthy.

A failure taxonomy for agent-written changes Tag each incident with the failure class and the earliest control that should have caught it.

When an agent causes an incident Containment, attribution through provenance, and a postmortem that names the failed control.

The evidence bundle The pull request contract whose provenance fields carry the session join key.

Metrics frameworks Canonical definitions of accepted change rate and cost per accepted change.

Keeping an agent-written codebase healthy The trend signals that sit beside these run-level panels.

Agent telemetry tools The ecosystem shelf: trace platforms, usage trackers, and gateways that consume this data.