# Evaluating Agents, Prompts and CLAUDE.md Changes: promptfoo, Inspect SWE, Langfuse and Braintrust

URL: https://developertoolkit.ai/en/ecosystem/observability/evals-coding-agents/
Description: A/B test a CLAUDE.md or AGENTS.md change on a fixed task set with promptfoo or Inspect SWE, then trace failing sessions in Langfuse or Braintrust.

Evaluating a coding agent means running a fixed set of real tasks against two configurations and comparing outcomes that code can check. promptfoo runs Claude Code and Codex through their SDKs, Inspect SWE runs them inside Docker sandboxes, and Langfuse or Braintrust trace each session so that a failed task can be explained.

Someone on your team opens a pull request that cuts `CLAUDE.md` from 400 lines to 120 and adds two new rules. The review thread is six opinions and no data: one person says the agent "felt sharper" after a day with the new file, another says it stopped running the tests. You are about to merge a change that every agent session in the repository will read, and nobody can say whether it helps. This page is for the **developer** who builds the evaluation and the **tech lead** who decides whether the rules change ships.

## What you get from an eval setup for agent rules

- **A promptfoo config that A/B tests two `CLAUDE.md` variants** on the same tasks, the same commit and the same pinned model, with deterministic checks (tests pass, diff stays in scope, tests were run) before any model-graded score.
- **The same experiment for Codex and `AGENTS.md`**, and the route for Cursor, where promptfoo has no provider.
- **An Inspect SWE task** for when you want every run in a disposable Docker sandbox with the model key kept on the host.
- **A decision rule** for reading the results, so a rules change is merged on evidence rather than on a feeling.
- **Where trace platforms fit**: Langfuse, Braintrust, Opik and Phoenix explain *why* a task failed; they do not replace the fixed task set.
- **Three copy-paste prompts** and a failure-modes table for the ways these experiments quietly measure nothing.

Every command and config key below was checked on 2026-09-26 against promptfoo 0.123.1, inspect-ai 0.3.269 with inspect-swe 0.2.71, Claude Code 2.1.283 and Codex CLI 0.157.1. For the general method of building an eval set to choose a *model*, see [reading benchmarks and building your own evals](/en/ecosystem/models/benchmarks/); this page is about evaluating changes to *your harness*: rules files, prompts, skills and plugins.

## Which eval tool answers which question?

Start from the question. The first three rows run the agent; the rest observe it or test something you ship.

| Question | Tool | Runs the agent? | Where it runs |
| --- | --- | --- | --- |
| Is the new `CLAUDE.md` or `AGENTS.md` better on our tasks? | promptfoo (`anthropic:claude-agent-sdk`, `openai:codex-sdk`) | Yes, through the vendor SDK | Your machine or a CI runner |
| Same, but every run isolated in a container, several agents in one harness | Inspect AI + Inspect SWE (`claude_code()`, `codex_cli()`) | Yes, inside a sandbox; model calls proxied through Inspect | Docker or Kubernetes |
| Does this plugin or skill help compared with no plugin? | `claude plugin eval` (Claude Code 2.1.283) | Yes, with a no-plugin baseline arm | Your machine |
| Why did this session fail, and what did it cost? | Langfuse plugin, Braintrust `bt trace`, Opik | No, traces real sessions | SaaS or self-hosted |
| Local trace UI for OpenTelemetry data, no account | Arize Phoenix | No | Your machine |
| Does the LLM feature *we ship* regress or leak? | DeepEval, garak, OpenLLMetry | No, tests your app | CI, your app |

Adoption as of 2026-09-26, from GitHub and the package registries: [promptfoo](https://github.com/promptfoo/promptfoo) 25.5k stars (npm 0.123.1; its README says it "is now part of OpenAI" and remains MIT-licensed), [Langfuse](https://github.com/langfuse/langfuse) 35.1k (the Claude Code plugin repository itself: 24), [Opik](https://github.com/comet-ml/opik) 22.2k, [DeepEval](https://github.com/confident-ai/deepeval) 18.4k, [Arize Phoenix](https://github.com/Arize-ai/phoenix) 11.6k, [garak](https://github.com/NVIDIA/garak) 9.4k, [OpenLLMetry](https://github.com/traceloop/openllmetry) 7.4k, [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai) 2.9k and [Inspect SWE](https://github.com/meridianlabs-ai/inspect_swe) 33. Braintrust's [coding-agent plugin monorepo](https://github.com/braintrustdata/braintrust-coding-agent-plugins) had 2 stars (npm `braintrust` 3.35.0). The platforms are established; their coding-agent plugins are months old.

## A/B test a CLAUDE.md change with promptfoo

The experiment has one variable. Both arms get the same repository commit, the same tasks, the same model and the same tool permissions; only the rules file differs. promptfoo runs each task through each arm, runs your checks, and shows the pass rate, cost and latency per arm side by side.

1. **Build a fixed task set of 10 to 20 real tasks.** Take them from merged pull requests that came with tests: a bug fix, a small feature, a refactor, a test-writing task. Each task needs a prompt a developer would really write and a check that code can run. Freeze the list; changing tasks between runs breaks the comparison.

2. **Create two worktrees outside the repository**, both at the same commit, and commit the proposed rules into the second one. Claude Code reads `CLAUDE.md` files from parent directories too, so a worktree inside your repository would also load the current file into the "proposed" arm. In a terminal, from the repository root:

   ```bash
   BASE=$(git rev-parse origin/main)
   mkdir -p ../rules-ab
   git worktree add --detach ../rules-ab/current "$BASE"
   git worktree add --detach ../rules-ab/proposed "$BASE"
   git show origin/trim-claude-md:CLAUDE.md > ../rules-ab/proposed/CLAUDE.md
   git -C ../rules-ab/proposed add CLAUDE.md && git -C ../rules-ab/proposed commit -qm "eval: proposed CLAUDE.md"
   (cd ../rules-ab/current && npm ci) && (cd ../rules-ab/proposed && npm ci)
   cd ../rules-ab && npm install --save-dev promptfoo@0.123.1 @anthropic-ai/claude-agent-sdk
   ```

   `trim-claude-md` stands for the branch of the pull request under review. Committing the proposed file lets the reset hook return each arm to a clean state with `git reset --hard` without losing the rules under test.

3. **Write the config.** Save it as `../rules-ab/promptfooconfig.yaml`. The two providers are identical except for `label` and `working_dir`:

   ```yaml title="rules-ab/promptfooconfig.yaml"
   description: CLAUDE.md A/B on a fixed task set
   prompts:
     - '{{task}}'
   providers:
     - id: anthropic:claude-agent-sdk
       label: current
       config:
         working_dir: ./current
         setting_sources: ['project']      # load CLAUDE.md from working_dir; no user-level files
         model: claude-opus-5-5            # pin it: a default that moves mid-experiment ruins the A/B
         tools: ['Read', 'Grep', 'Glob', 'Write', 'Edit', 'Bash']   # what the agent can see
         permission_mode: dontAsk          # anything that would prompt is denied, never prompted
         append_allowed_tools: ['Write', 'Edit', 'Bash(npm test:*)', 'Bash(npx tsc:*)']
         disallowed_tools: ['WebFetch', 'WebSearch']
         max_turns: 40
         max_budget_usd: 3
     - id: anthropic:claude-agent-sdk
       label: proposed
       config:
         working_dir: ./proposed
         setting_sources: ['project']
         model: claude-opus-5-5
         tools: ['Read', 'Grep', 'Glob', 'Write', 'Edit', 'Bash']   # what the agent can see
         permission_mode: dontAsk          # anything that would prompt is denied, never prompted
         append_allowed_tools: ['Write', 'Edit', 'Bash(npm test:*)', 'Bash(npx tsc:*)']
         disallowed_tools: ['WebFetch', 'WebSearch']
         max_turns: 40
         max_budget_usd: 3
   extensions:
     - file://reset.js:extensionHook
   defaultTest:
     options:
       provider: openai:responses:gpt-6-sol   # the judge comes from another vendor than the agent
     assert:
       - type: javascript
         value: file://checks.js:testsPass
         metric: tests_pass
       - type: javascript
         metric: ran_tests
         value: |
           const calls = context.providerResponse?.metadata?.toolCalls || [];
           return calls.some(t => t.name === 'Bash' && t.input?.command?.includes('npm test'));
       - type: llm-rubric
         metric: conventions
         value: >-
           The final message states what changed and why, and names the tests it ran.
           You see only this message, not the diff.
   tests:
     - description: invoice rounding bug (from PR 1841)
       vars:
         task: >-
           Invoice totals are one cent off when a line has quantity 3 and unit price 0.10.
           Fix it in src/billing and add a regression test.
       assert:
         - type: javascript
           metric: in_scope
           value: file://checks.js:diffInScope
           config: { allowedPrefixes: ['src/billing/', 'test/billing/'] }
     # ...9 to 19 more tasks in the same shape
   ```

   `setting_sources` is off by default in the promptfoo provider, so without it neither arm reads any `CLAUDE.md` and the two arms are the same experiment twice. Set `tools` explicitly: with a `working_dir` and no `tools`, promptfoo 0.123.1 derives the tool list from the allowed-tools entries, and a pattern such as `Bash(npm test:*)` is a permission rule, not a tool name. `tools` decides what the agent can call; `append_allowed_tools` decides what runs without a prompt; `dontAsk` refuses what would otherwise prompt. Take the current model ID from the [model comparison hub](/en/ecosystem/models/) when you run this.

4. **Add the checks and the reset hook** next to the config. The checks look at the arm's working directory after the agent finishes; the hook puts both arms back to their commit after every test.

   ```js title="rules-ab/checks.js"
   const { execFileSync } = require('node:child_process');
   const path = require('node:path');

   const dirOf = (context) => path.join(__dirname, context.provider.label);
   const run = (cmd, args, cwd) => {
     try {
       return { ok: true, out: execFileSync(cmd, args, { cwd, encoding: 'utf8', stdio: 'pipe', timeout: 600000 }) };
     } catch (err) {
       return { ok: false, out: `${err.stdout || ''}${err.stderr || ''}` };
     }
   };

   module.exports.testsPass = (output, context) => {
     const r = run('npm', ['test', '--silent'], dirOf(context));
     return { pass: r.ok, score: r.ok ? 1 : 0, reason: r.ok ? 'npm test passed' : r.out.slice(-800) };
   };

   module.exports.diffInScope = (output, context) => {
     const cwd = dirOf(context);
     const changed = run('git', ['diff', '--name-only', 'HEAD'], cwd).out;
     const added = run('git', ['ls-files', '--others', '--exclude-standard'], cwd).out;
     const files = `${changed}\n${added}`.split('\n').filter(Boolean);
     const outside = files.filter((f) => !context.config.allowedPrefixes.some((p) => f.startsWith(p)));
     return { pass: outside.length === 0, score: outside.length ? 0 : 1, reason: outside.length ? `outside scope: ${outside.join(', ')}` : `${files.length} files, all in scope` };
   };
   ```

   ```js title="rules-ab/reset.js"
   const { execFileSync } = require('node:child_process');
   const path = require('node:path');

   function reset() {
     for (const arm of ['current', 'proposed']) {
       const cwd = path.join(__dirname, arm);
       execFileSync('git', ['reset', '--hard', '--quiet', 'HEAD'], { cwd });
       execFileSync('git', ['clean', '-fdq'], { cwd });   // keeps ignored files such as node_modules
     }
   }

   module.exports.extensionHook = async (hookName) => {
     if (hookName === 'beforeAll' || hookName === 'afterEach') reset();
   };
   ```

5. **Run it serially, three times per task, with the cache off.** Serial, because both arms share the reset hook; three repeats, because one agent run per task is too noisy to compare; no cache, because a cached answer is not a new run.

   ```bash
   cd ../rules-ab
   # Load ANTHROPIC_API_KEY (the agent) and OPENAI_API_KEY (the judge) into the environment
   # from your secret manager; never type a key on the command line. To use your local
   # Claude Code login instead of an API key, add apiKeyRequired: false to both providers.
   npx promptfoo eval -j 1 --repeat 3 --no-cache -o results.json
   npx promptfoo view
   ```

   `promptfoo eval` exits with code `100` when at least one test fails, so the same command works as a CI gate later. `promptfoo view` opens the matrix in a browser: one column per arm, one row per task, with the named metrics (`tests_pass`, `ran_tests`, `conventions`, `in_scope`) and cost per cell.

**The agent runs real commands on the machine that runs the eval**

The config auto-approves file edits and two command prefixes only, and disallows the WebFetch and WebSearch tools. `permission_mode: dontAsk` refuses every other call that would prompt instead of waiting for an approval nobody will give. Even so, run the experiment on a disposable machine or container that holds a copy of the repository and the model keys, and nothing else: no cloud credentials, no SSH keys, no production `.env` files.

## Run the same experiment in Claude Code, Codex and Cursor

The task set, the checks and the decision rule do not change. What changes is the provider and the rules file each agent reads.

**Claude Code**

    Use the config above. The provider IDs are `anthropic:claude-agent-sdk` and its alias `anthropic:claude-code`; `anthropic:claude-code-sdk` does not exist. promptfoo 0.123.1 installs `@anthropic-ai/claude-agent-sdk` 0.3.263 as an optional dependency; installing it explicitly, as in step 2, gets the current release (npm 0.3.283 on 2026-09-26) and covers installs that skip optional dependencies.

    To A/B a **skill or a plugin** rather than the rules file, Claude Code 2.1.283 has a built-in runner. It reads eval cases from the plugin's `evals/` directory and adds a no-plugin baseline arm automatically:

    ```bash
    claude plugin eval ./plugins/release-notes --runs 3 --threshold 0.8 --no-publish
    ```

    The first run in a new plugin directory asks you to confirm trust; in CI pass `--trust-plugin`, and only for plugins you would run yourself. `--threshold` makes the command exit `1` when any case scores below it, and `--max-cost-usd` sets a hard ceiling for the run. It runs the plugin's code as you, so evaluate only plugins you trust. The case format is covered on [building and distributing a plugin](/en/ecosystem/plugins/building-plugins/).

**Codex**

    Codex reads `AGENTS.md`, so the variants differ in that file. Swap the two providers for the Codex SDK provider; `openai:codex` is an alias. promptfoo 0.123.1 declares `@openai/codex-sdk` `^0.153.2` as an optional dependency (npm 0.157.1 on 2026-09-26).

    ```yaml title="rules-ab/promptfooconfig.yaml (providers only)"
    providers:
      - id: openai:codex-sdk
        label: current
        config:
          working_dir: ./current
          model: gpt-6-astra
          sandbox_mode: workspace-write     # writes stay inside working_dir
          approval_policy: never
          network_access_enabled: false
      - id: openai:codex-sdk
        label: proposed
        config:
          working_dir: ./proposed
          model: gpt-6-astra
          sandbox_mode: workspace-write
          approval_policy: never
          network_access_enabled: false
    ```

    Drop the `ran_tests` assertion: it reads Claude-style tool calls. The `testsPass` and `diffInScope` checks work unchanged. Since Codex 0.150.0 untrusted projects supply no project `AGENTS.md`; mark both worktree paths as trusted, and run the canary rule on the Codex arms before you trust a tie.

**Cursor**

    promptfoo 0.123.1 has no Cursor provider (checked against its provider index at tag `0.123.1`). Keep the task set and the checks, and run them through [Harbor](/en/ecosystem/models/benchmarks/)'s `cursor-cli` agent (Harbor 0.23.0, 2026-09-12), which installs the Cursor CLI in each task's container and drives it in print mode with auto-approval. Build two task directories that differ only in the rules files baked into the task image, then run both. Load `CURSOR_API_KEY` into the environment from your secret manager; Harbor passes it into the task container.

    ```bash
    harbor run -p evals/tasks-current  -a cursor-cli -m "$CURSOR_MODEL" -k 3 -n 4
    harbor run -p evals/tasks-proposed -a cursor-cli -m "$CURSOR_MODEL" -k 3 -n 4
    ```

    `CURSOR_MODEL` takes Harbor's `provider/model` form; Harbor passes the part after the slash to Cursor, so use a model name your Cursor plan offers. `-k 3` runs three attempts per task and `-n 4` runs four at once. Because Harbor auto-approves every command, the task container is the only boundary: keep credentials other than the Cursor key out of it, and restrict its outbound network to the Cursor API and the hosts the installer downloads from. Task layout, `oracle` and `nop` sanity runs and the other flags are on the benchmarks page.

    Cursor's docs (checked 2026-10-12) say project rules live in `.cursor/rules` as `.mdc` files (a plain `.md` file there is ignored), that Cursor supports `AGENTS.md` in the project root and subdirectories, and that rules apply to Agent chat only. This page did not test whether Cursor's print mode in Harbor loads them. Before the first run, add a canary rule to both arms, such as "end every final message with the word CANARY", and confirm it appears. A rules file that is not loaded produces a perfect tie.

## Run the A/B in a sandbox with Inspect SWE

Choose Inspect SWE when you want every run in a fresh container, or one harness for several agents: its `claude_code()` and `codex_cli()` agents run inside the sample's sandbox, and their model calls are proxied back through Inspect, so token limits, time limits and transcripts work as for any Inspect eval and the API key never enters the container.

```python title="rules_ab.py"
from pathlib import Path

from inspect_ai import Task, task
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import CORRECT, INCORRECT, Score, Target, accuracy, scorer, stderr
from inspect_ai.solver import Generate, TaskState, solver
from inspect_ai.util import sandbox
from inspect_swe import claude_code

@solver
def install_rules(rules_file: str):
    async def solve(state: TaskState, generate: Generate) -> TaskState:
        await sandbox().write_file("/repo/CLAUDE.md", Path(rules_file).read_text())
        return state
    return solve

@scorer(metrics=[accuracy(), stderr()])
def tests_pass():
    async def score(state: TaskState, target: Target) -> Score:
        result = await sandbox().exec(["npm", "test", "--silent"], cwd="/repo")
        return Score(value=CORRECT if result.success else INCORRECT, explanation=result.stderr[-800:])
    return score

@task
def rules_ab(rules: str = "rules/current.md") -> Task:
    return Task(
        dataset=json_dataset("tasks.jsonl"),        # one {"id", "input", "target"} per line
        setup=install_rules(rules),
        solver=claude_code(cwd="/repo", disallowed_tools=["WebSearch", "WebFetch"]),
        scorer=tests_pass(),
        sandbox=("docker", "compose.yaml"),         # image with the repository at /repo, deps installed, no secrets
        epochs=3,
    )
```

Run each arm, then compare them in the log viewer:

```bash
pip install inspect-ai==0.3.269 inspect-swe==0.2.71
inspect eval rules_ab.py --model anthropic/claude-opus-5-5 -T rules=rules/current.md
inspect eval rules_ab.py --model anthropic/claude-opus-5-5 -T rules=rules/proposed.md
inspect view
```

For Codex, import `codex_cli` and write `AGENTS.md` instead. In Inspect SWE 0.2.71, `codex_cli()` defaults to `sandbox_mode="danger-full-access"` (Codex's own sandbox off) and `web_search="live"`. The first is acceptable only because the Inspect container is the boundary, so keep that container free of credentials and block its outbound network: model calls already travel back through Inspect, so the agent needs no direct route out. Pass `web_search="disabled"` to both arms, the equivalent of `disallowed_tools=["WebSearch", "WebFetch"]` on the Claude Code side, so neither arm can look up the answer. The PyPI package installs with `pip install inspect-swe`; the README's own install line is the git development build.

## How do you decide from the results?

Write the rule before you run, so the result cannot be argued into the answer someone wanted.

| Result | Decision |
| --- | --- |
| `tests_pass` for proposed is at least current's, `in_scope` and `ran_tests` no worse, cost per task within 20% | Merge the rules change, and paste the per-arm table into the pull request |
| Proposed wins on some tasks and loses on others | Read the tasks where the arms disagree in all three repeats; those are signal, one-off flips are noise. Fix the rule that caused the loss and rerun |
| Pass rates equal, proposed cheaper or faster | Merge: a shorter file that keeps behaviour saves tokens on every session |
| Proposed worse on `tests_pass` in any task across all three repeats | Do not merge. Find the rule that the old file had and the new one lost |
| Only the `conventions` judge score differs | Treat it as a hint, not a verdict, until the judge is calibrated against your own labels ([model-graded checks](/en/workflows/test/model-graded-checks/)) |

Row 4 overrides row 1: a consistent loss on any task blocks the merge even when the overall pass rate is equal or higher.

Deterministic checks outrank the judge. A judge that likes longer final messages will reward a verbose rules file, and that is how a regression gets merged with a better score.

## Where trace platforms fit in the loop

A fixed task set tells you *whether* a change helps. A trace tells you *why* one task failed: which file the agent read first, which rule it quoted, where it gave up. Add tracing when the eval shows a loss you cannot explain from the final message.

**Langfuse**

    The official plugin traces prompts, turns, model generations with token usage and cost, tool calls, skills and subagents for the `claude` CLI and the desktop app in Code mode. It needs `uv` on `PATH`.

    ```bash
    claude plugin marketplace add langfuse/Claude-Observability-Plugin
    claude plugin install langfuse-observability@langfuse-observability
    ```

    Then configure the keys from inside a session with `/plugin configure langfuse-observability@langfuse-observability`; the secret key goes to the OS keychain, not a file. Restart Claude Code. If no trace appears, read `~/.claude/state/langfuse_hook.log`. Langfuse can then run datasets and LLM-as-judge evaluations over the traced sessions.

**Braintrust**

    The `bt` CLI installs a plugin and a local tracing daemon for Claude Code, Codex, OpenCode, Pi, Grok and Antigravity. Each session becomes one trace with turn, model-call, tool and subagent spans.

    ```bash
    curl -fsSL https://bt.dev/cli/install.sh | bash   # Windows and mise installs: README of braintrustdata/bt
    bt login
    bt trace enable claude --project rules-ab
    bt trace doctor claude
    bt trace run --project rules-ab --additional-metadata '{"arm":"proposed"}' claude -- -p "Fix the invoice rounding bug in src/billing"
    ```

    The plugin does not redact prompts, tool input and output, or system prompts locally. Configure Braintrust's redaction before you enable tracing on a real repository.

**Opik and Phoenix**

    **Opik** (Comet, Apache-2.0, self-hostable) connects to Claude Code, Codex, Cursor and others over MCP, so you can ask the agent to score its own recent traces:

    ```bash
    pip install opik && opik configure
    uvx opik mcp configure
    ```

    **Arize Phoenix** gives a local OpenTelemetry trace and eval UI with no account: `pip install arize-phoenix && phoenix serve`. Wiring Claude Code's own sessions into Phoenix could not be verified on 2026-09-26; use it for traces your application emits.

Context cost: promptfoo, Inspect and `bt trace run` run outside your interactive sessions and add nothing to their context. The Langfuse and Braintrust plugins work through hooks; run `claude plugin details <plugin>` after installing to see the projected token cost of any component they add. Opik's MCP server adds tool definitions to every session that loads it, so measure it with `/context` before and after.

For your own application's LLM features rather than the coding agent, the tools differ: **DeepEval** (`deepeval test run test_chatbot.py`) gates a feature in CI with pytest-style assertions, **garak** (`pip install garak`) probes a model endpoint for jailbreaks, prompt injection and leakage, and **OpenLLMetry** (`pip install traceloop-sdk`, then `Traceloop.init()`) instruments the app with OpenTelemetry. None of the three traces a coding agent. The legacy `openai/evals` repository now points to evals in the OpenAI dashboard; its PyPI package `evals` last shipped in May 2024.

## Copy-paste prompts for agent evals

**Copy-paste prompt: build the fixed task set from merged pull requests**

```text
Read the last 60 merged pull requests in this repository with `gh pr list --state merged --limit 60 --json number,title,files,body`.

Pick 15 that (1) changed code and tests together, (2) touched at most 6 files, and (3) cover a mix: at least 5 bug fixes, 4 small features, 3 refactors and 3 test-only tasks. Skip anything that touched migrations, CI config or dependencies.

For each, write one line of JSON with: id (pr-<number>), the task prompt a developer would have typed before the work existed (no hints from the final diff), the directories the change may touch as allowedPrefixes, and the test command that proves it. Write them to evals/tasks.jsonl. Do not modify any other file.
```

**Copy-paste prompt: turn a rules change into an A/B experiment**

```text
Compare CLAUDE.md on main with the version in this branch. List every rule that was added, removed or reworded, and for each one name the behaviour it should change.

For each behaviour, propose one deterministic check that a script can run in the working directory after the agent finishes (a test command, a git diff condition, a grep over changed files) and say which tasks in evals/tasks.jsonl would exercise it. If no task exercises a rule, draft one new task for it.

Output a table (rule, expected behaviour, check, tasks) and the JavaScript for any new check in the style of rules-ab/checks.js. Do not run the eval.
```

**Copy-paste prompt: read the promptfoo results and recommend a decision**

```text
Read rules-ab/results.json from a promptfoo run with two providers labelled "current" and "proposed", three repeats per task.

Report per arm: pass rate for each named metric (tests_pass, ran_tests, conventions, and in_scope where present), mean cost and mean latency per task. Then list every task where the arms disagree on tests_pass in all three repeats, and separately the tasks that flip between repeats.

Apply this rule: merge only if proposed is no worse on tests_pass, in_scope and ran_tests, and its mean cost is within 20% of current. State the decision, the evidence for it, and for each consistent loss quote the part of the proposed CLAUDE.md most likely responsible. Do not treat a conventions-only difference as decisive.
```

## How do you know the eval itself works?

An eval that cannot fail is a tautology with a cost line. Before you trust a result, prove the harness can detect a difference:

- **Canary rule.** Add a harmless rule to one arm only ("end your final message with CANARY-B") and run one task. If the marker does not appear, the rules file is not being loaded.
- **Known-bad variant.** Run once with a proposed file that tells the agent not to run tests. `ran_tests` and usually `tests_pass` must drop. If they do not, the checks are too weak; see [oracle strength](/en/workflows/test/oracle-strength/).
- **Judge from another vendor.** The agent arm never grades itself: the config above sends `llm-rubric` to a different vendor's model. Calibrate it on 20 hand-labelled outputs before it can block a merge.
- **Rerun on every model change.** A rules file tuned on one model can regress on the next. The [new-model playbook](/en/ecosystem/models/new-model-playbook/) reruns this task set as its first step.

Ownership: the developer who proposes the rules change runs the A/B and attaches the per-arm table to the pull request, as part of its [evidence bundle](/en/workflows/ship/evidence-bundle/). The tech lead owns the task set, reviews the table rather than the prose of the new file, and signs off the merge.

## What breaks when you evaluate agent rules?

| Symptom | Cause | Recovery |
| --- | --- | --- |
| Both arms score identically on every task | `setting_sources` missing, so neither arm reads `CLAUDE.md` | Add `setting_sources: ['project']`; confirm with the canary rule |
| The proposed arm still follows a rule you deleted | Worktree inside the repository: Claude Code also loads the parent directory's `CLAUDE.md` | Put both worktrees outside the repository, as in step 2 |
| A personal preference changes results on one laptop | `user` in `setting_sources` loads `~/.claude/CLAUDE.md` | Use `['project']` only |
| The second run finishes in seconds with the same scores | promptfoo served cached responses | Always pass `--no-cache` for agent evals |
| Tests pass in one arm because of the other arm's edits | Parallel runs or no reset between tests | `-j 1` and the `reset.js` hook |
| Scores flip between runs of the same task | One run per task, or fewer than 10 tasks | `--repeat 3` and more tasks; decide on consistent disagreements only |
| The Codex arm ignores `AGENTS.md` | Worktree path not trusted (Codex 0.150.0 and later) | Mark both eval paths trusted; confirm with the canary rule |
| One task burns the budget | A loop of failing tool calls | `max_turns` and `max_budget_usd` per provider; `--max-cost-usd` for `claude plugin eval` |
| Results shift between Monday and Friday | The default model changed | Pin `model` in both arms, and rerun both arms together |
| Traces show API keys and file contents in the vendor UI | Braintrust and Langfuse capture prompts and tool output | Configure redaction first; trace a sanitised fixture repository |

**Traps in agent eval tooling**

- The promptfoo provider IDs are `anthropic:claude-agent-sdk` (alias `anthropic:claude-code`) and `openai:codex-sdk` (alias `openai:codex`). `anthropic:claude-code-sdk` is wrong, and "Claude Code SDK" is the old name of the Claude Agent SDK.
- `pip install openai-evals` is not a package; the legacy package is `evals`, and OpenAI now points to evals in its dashboard.
- Inspect SWE's README installs the git development build; `pip install inspect-swe` gets the PyPI release.
- The Langfuse, Braintrust and Opik coding-agent plugins are young (24, 2 and 23 stars on 2026-09-26) even though the platforms are popular. Pin their versions and check them after every Claude Code upgrade.
- garak scans model endpoints, not agent configuration or skills. For those, use Snyk Agent Scan from [agent security gates](/en/ecosystem/observability/agent-security-gates/).
- Do not start a new project on Helicone: it has been in maintenance mode since March 2026 (secondary sources: the Helicone and Mintlify acquisition posts).

## Where to go next with agent evals

To run this task set on every harness change rather than once per PR, see [continuous evals](/en/developer-scorecard-guide/continuous-evals/).

- [Continuous evals](/en/developer-scorecard-guide/continuous-evals/) — Regression-test the agent harness on every change, not once per PR.
- [Reading benchmarks and building your own evals](/en/ecosystem/models/benchmarks/) — Public benchmarks, Harbor, and a 30-task eval set that picks your model.
- [Model-graded checks](/en/workflows/test/model-graded-checks/) — Calibrate a rubric judge before it can block a merge.
- [When a new model ships](/en/ecosystem/models/new-model-playbook/) — Rerun this task set first, then re-tune effort and rules.
- [Telemetry and analytics](/en/ecosystem/observability/agent-telemetry/) — OpenTelemetry for Claude Code and Codex, joined to pull requests.

Before you change the rules file, read [how to write and trim `CLAUDE.md`](/en/claude-code/build/memory-system/); this page tells you whether the trimmed version still works. For the production side of the same data, see [observing the agents](/en/workflows/operate/agent-observability/). The section hub, [measuring agentic engineering](/en/ecosystem/observability/), puts evals beside telemetry, cost tracking, review bots and security gates.
