Skip to content
AI Developer Toolkit

Find your level — free Buy the Toolkit Buy Toolkit Buy the Toolkit Buy Toolkit Open the guides Read
Navigation menu
Buy the Toolkit Find your level — free Buy the Toolkit Open the guides Sign in

Continuous evals — regression-test the agent harness in CI

PL

Level 2 How-to Updated

Continuous evals are a standing regression suite for the agent harness: real repository tasks that CI runs headless on every change to CLAUDE.md, AGENTS.md, rules, skills, hooks, the pinned CLI version or the model, and weekly on main. Deterministic checks grade each run, and a merge is blocked when repeated pass rates fall.

Last week someone trimmed CLAUDE.md from 300 lines to 90 and the pull request looked like pure cleanup. Two days later agent PRs stopped running the migration check, and the first person to notice was the on-call engineer. Nothing in the product test suite could have caught it, because the product code never changed. This page is for the developer who builds the suite and the tech lead who owns its baseline and signs off harness changes.

Scorecard Q17: How do you regression-test your agent configuration (CLAUDE.md, rules, skills, hooks)?

Maximum-score answer (3 points): a continuous CI eval workflow (.github/workflows/agent-evals.yml) that runs on configuration and skill pull requests, with production incidents added as regression tests.

What a continuous eval pipeline gives you

Section titled “What a continuous eval pipeline gives you”
  • A case format (prompt.md, base-commit, check.sh) that grades an agent run with code, using tests the agent never sees.
  • A runner that replays each case several times against the harness under test, for Claude Code, Codex or Cursor.
  • A gate that blocks the merge when a must-pass case fails or the pass rate drops below the reviewed baseline.
  • A GitHub Actions workflow with least-privilege settings, triggered by harness changes, CLI bumps and a weekly schedule.
  • An incident-to-case procedure, so a failure that reached production is tested on every future harness change.

How is a continuous eval different from a one-off A/B test?

Section titled “How is a continuous eval different from a one-off A/B test?”

An A/B test answers one question once: is this proposed CLAUDE.md better than the current one? Evaluating agents and CLAUDE.md changes covers that experiment with promptfoo, Inspect SWE and Harbor. A continuous eval answers a different question every day: did anything that steers the agent get worse since the last accepted baseline?

The difference shows up in three places. The suite runs without anyone deciding to run it. It compares against a baseline a person accepted, not against a second arm. And it grows: every incident adds a case, and cases that no longer discriminate are retired.

What should trigger the suite?

Section titled “What should trigger the suite?”

Most harness regressions arrive without a code change a reviewer would flag. Trigger on each source of drift:

TriggerWhat it catchesHow to wire it
A pull request that edits CLAUDE.md, AGENTS.md, .claude/, .agents/, .codex/ or .cursor/Rules, skills, hooks or subagents that stopped doing their jobpull_request with paths:
A bump of the pinned Claude Code or Codex CLI versionChanged defaults, tools or loading behaviour in a new releaseKeep the version in the workflow file, which is itself in paths:
A change of the pinned modelA rule tuned on one model regressing on the nextPin the model in the agent script; change it in a pull request
Nothing at all, weeklyServer-side model and service changes that ship with no commitschedule: on main
An incidentThe exact failure that escapedA new must-pass case, merged with the postmortem

Pin the CLI versions and the model in the eval job. An unpinned npm install -g @anthropic-ai/claude-code turns every release into an unreviewed harness change, and a failing run can no longer tell you whether your rules or the tool moved.

Build the case suite

Section titled “Build the case suite”

  1. Collect 20 to 50 real tasks from the last few months. Take them from merged pull requests and from tasks the agent got wrong. Mix the work your team repeats: an authenticated endpoint, a schema migration, a UI component that must use design tokens, a fix in an async worker.

  2. Give each case its own directory. evals/cases/<id>/ holds prompt.md (the task as a developer would type it), base-commit (the commit the task started from), check.sh (the grader), and any hidden test files the grader copies in.

  3. Grade with code, not with opinion. check.sh receives the worktree path, the agent’s log path and the SHA of the harness commit the run started from. It exits 0 for a pass. Prefer tests, the type checker, the linter and scope checks; add a model-graded check only for what code cannot decide, and only after it is calibrated.

  4. Mark the cases that must never fail. An empty must-pass file in the case directory tells the gate that one failed run blocks the merge. Every incident case and the canary case get one.

  5. Add a canary case. It asks a question only the harness can answer, such as “Which command must pass before you declare a task done in this repository? Reply with the command only.” Its check greps the log for the command your CLAUDE.md or AGENTS.md names. When the canary fails, the harness was not loaded and every other result is meaningless.

A grader for a case that adds an authenticated settings route:

evals/cases/authz-settings-route/check.sh
#!/usr/bin/env bash
# Usage: check.sh <worktree> <agent-log> <base-sha>. Exit 0 = pass.
set -euo pipefail
here=$(cd "$(dirname "$0")" && pwd)
base=$3   # the "harness under test" commit the run started from
cd "$1"
test -f src/routes/settings.ts
grep -q 'requireAuth' src/routes/settings.ts
# A test the agent never saw: it lives in the case directory, not in the repository.
cp "$here/settings.hidden.test.ts" tests/routes/
npm run typecheck
npx vitest run tests/routes/settings.hidden.test.ts
# Scope: the agent may touch routes and their tests, nothing else, committed or not.
outside=$( { git diff --name-only --no-renames "$base" -- .; git ls-files --others --exclude-standard; } \
  | grep -vE '^(src|tests)/routes/' || true)
if [ -n "$outside" ]; then
  echo "change outside src/routes or tests/routes:"; echo "$outside"; exit 1
fi

The hidden test is what makes the grade trustworthy. An agent that can read the test can write code that passes it and nothing more; protecting the oracle covers the other ways a check leaks.

Write the runner

Section titled “Write the runner”

The runner checks out each case’s base commit in a worktree outside the repository, overlays the harness from the checkout under test, commits it so the agent’s changes are the only diff, runs the agent, and grades the result. Save it as evals/run.sh:

evals/run.sh
#!/usr/bin/env bash
# Run every case EVAL_RUNS times with the harness in this checkout.
set -uo pipefail
root=$(pwd); runs=${EVAL_RUNS:-3}; agent=${EVAL_AGENT:-claude}
codex_src=${CODEX_HOME:-$HOME/.codex}
out="$root/evals/out"; mkdir -p "$out"; : > "$out/results.tsv"
for case in "$root"/evals/cases/*/; do
  id=$(basename "$case")
  for n in $(seq 1 "$runs"); do
    dir=$(mktemp -d); log="$out/$id.$n.log"
    git worktree add --detach "$dir" "$(cat "$case/base-commit")" >/dev/null 2>&1
    # The base commit carries the OLD harness: replace it with the one under test.
    for f in CLAUDE.md AGENTS.md .claude .agents .codex .cursor; do
      rm -rf "${dir:?}/$f"
      if [ -e "$root/$f" ]; then cp -R "$root/$f" "$dir/$f"; fi
    done
    rm -rf "${dir:?}/evals"   # the agent never sees cases or graders
    git -C "$dir" add -A
    git -C "$dir" -c user.name=evals -c user.email=evals@localhost commit -qm "harness under test" --allow-empty
    base=$(git -C "$dir" rev-parse HEAD)
    if [ "$agent" = codex ]; then   # a throwaway Codex home: trust entries never reach ~/.codex
      export CODEX_HOME=$(mktemp -d); cp "$codex_src/auth.json" "$CODEX_HOME/"
    fi
    (cd "$dir" && npm ci --no-audit --prefer-offline >/dev/null 2>&1)
    (cd "$dir" && timeout 900 bash "$root/evals/agent-$agent.sh" "$(cat "$case/prompt.md")") > "$log" 2>&1
    if bash "$case/check.sh" "$dir" "$log" "$base" >> "$log" 2>&1; then r=pass; else r=fail; fi
    printf '%s\t%s\t%s\n' "$id" "$n" "$r" | tee -a "$out/results.tsv"
    git worktree remove --force "$dir"
    if [ "$agent" = codex ]; then rm -rf "$CODEX_HOME"; fi
  done
done

Three runs per case is the minimum. Agents are not deterministic, so one run of one case is a sample, not a verdict. The worktree sits in a temporary directory because Claude Code also loads CLAUDE.md files from parent directories, and a worktree inside the repository would read the old file as well as the new one.

The agent line is the only part that differs per tool. Each script receives the prompt as $1 and runs in the worktree:

evals/agent-claude.sh
#!/usr/bin/env bash
# Loads project settings, CLAUDE.md, skills and hooks: the harness under test.
exec claude -p "$1" \
  --model "${EVAL_MODEL:-claude-opus-5-5}" \
  --setting-sources project \
  --permission-mode acceptEdits \
  --allowedTools "Bash(npm run *)" "Bash(npx vitest *)" "Bash(git diff *)" \
  --max-budget-usd 3 \
  --no-session-persistence \
  --output-format json

Flags checked against the Claude Code CLI reference and claude --help for 2.1.283 (the latest channel on 2026-09-26). Do not add --bare: the headless docs recommend it for scripted calls, but it skips CLAUDE.md auto-discovery, settings and plugin hooks, plugin sync and auto-discovered MCP config (skills still resolve only when invoked by name); from v2.1.286 it also sends no system reminders and starts no background tasks. That is most of what this suite exists to test. --setting-sources project keeps a developer’s personal ~/.claude settings out of the result. --max-budget-usd stops a case that loops on failing tool calls.

To evaluate one skill in isolation, Claude Code also ships claude plugin eval (since week 37, 2026-09-07), which works on plugin and skill folders and adds a no-plugin baseline arm. claude plugin eval init --bare <name>, run in the skill folder, writes a case template: evals/<name>/prompt.md and evals/<name>/graders/criteria.md (the eval dir defaults to evals/; change it with --eval-dir; checked against claude plugin eval init --help 2.1.283, the latest channel on 2026-09-26). Its default grader model is Haiku, so treat LLM-graded scores as hints until you calibrate them.

Gate the merge on repeated pass rates

Section titled “Gate the merge on repeated pass rates”

The gate reads evals/out/results.tsv and applies two rules. A must-pass case that fails any run blocks the merge. The overall pass rate may not fall more than five points below evals/baseline-rate, a one-line file the tech lead updates in a reviewed pull request.

evals/gate.sh
#!/usr/bin/env bash
set -euo pipefail
res=evals/out/results.tsv; block=0
for id in $(awk -F'\t' '$3=="fail"{print $1}' "$res" | sort -u); do
  if [ -f "evals/cases/$id/must-pass" ]; then echo "BLOCK: must-pass case $id failed a run"; block=1; fi
done
rate=$(awk -F'\t' '{t++; if ($3=="pass") p++} END {printf "%d", p*100/t}' "$res")
base=$(cat evals/baseline-rate)
echo "pass rate ${rate}% against baseline ${base}%"
if [ "$rate" -lt $((base - 5)) ]; then echo "BLOCK: pass rate fell more than 5 points"; block=1; fi
exit "$block"

The five-point tolerance is a starting value, not a law. With 30 cases and three runs, one flipped run moves the rate by about one point, so a smaller tolerance turns noise into blocked merges. Tighten it as the suite grows. The baseline never moves automatically: an automatic update would quietly accept every regression the suite exists to catch.

Run the suite in GitHub Actions

Section titled “Run the suite in GitHub Actions”
.github/workflows/agent-evals.yml
name: Agent evals
on:
  pull_request:
    paths:
      - '**/CLAUDE.md'
      - '**/AGENTS.md'
      - '.claude/**'
      - '.agents/**'
      - '.codex/**'
      - '.cursor/**'
      - 'evals/**'
      - '.github/workflows/agent-evals.yml'
  schedule:
    - cron: '17 4 * * 1'   # Mondays: catches model and service drift with no commit
  workflow_dispatch:


permissions:
  contents: read


concurrency:
  group: agent-evals-${{ github.ref }}
  cancel-in-progress: true


jobs:
  evals:
    # Fork pull requests get no secrets; never evaluate a fork's harness with your key.
    if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 120
    env:
      CLAUDE_CODE_VERSION: 2.1.283   # bump in a pull request: the bump runs the suite
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0             # cases pin base commits from history
          persist-credentials: false
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}"
      - name: Run the suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.EVALS_ANTHROPIC_API_KEY }}
          EVAL_AGENT: claude
          EVAL_RUNS: 3
        run: bash evals/run.sh
      - name: Gate
        run: bash evals/gate.sh
      - name: Keep the logs
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: agent-eval-results
          path: evals/out/

For Codex, install @openai/codex@0.157.1 instead, log in with printenv OPENAI_API_KEY | codex login --with-api-key in a step that has OPENAI_API_KEY from a secret, and set EVAL_AGENT: codex. Make the evals job a required status check on main, so a blocked gate blocks the merge.

Turn every incident into a regression case

Section titled “Turn every incident into a regression case”

An incident that the harness could have prevented becomes a case before the postmortem closes. The team that owned the incident writes it, because they know what the agent should have done.

  1. Name the failed control. Was it a missing rule, a rule the agent ignored, a skill that did not load, or a hook that let a command through? The postmortem in when an agent causes an incident records this.

  2. Reproduce the task, not the outcome. Set base-commit to the commit before the bad change, and write prompt.md as the task the agent was actually given.

  3. Write a check that fails on the incident’s diff. Run check.sh against the commit that caused the incident and confirm it exits non-zero. A check that passes on the known-bad change cannot catch the next one.

  4. Fix the harness in the same pull request. The new case fails on the old harness and passes on the fixed one; the eval job shows both.

  5. Add must-pass and merge with the postmortem. The incident ticket closes when the case is on main.

Copy-paste prompts for continuous evals

Section titled “Copy-paste prompts for continuous evals”

BAD_SHA is the commit that caused the incident.

How do you know the suite still catches regressions?

Section titled “How do you know the suite still catches regressions?”

A suite that always passes looks like safety and proves nothing. Four checks keep it honest, none of which needs anyone to read agent diffs:

  • The canary passes on every run. If it fails, the harness was not loaded; fix the runner before you read any other result.
  • A known-bad harness fails the gate. Once a quarter, open a throwaway pull request that deletes the CLAUDE.md lines about tests and security. The gate must block it. If it does not, the checks are too weak; how strong is your oracle shows how to measure that.
  • Every incident case fails on the incident’s commit. Step 3 of the incident procedure proves it once; the rule stops weak cases from entering the suite.
  • Saturated cases are retired. A case that every harness variant has passed for months no longer discriminates. Replace it with a newer task, preferably one the agent got wrong.

Sign-off is split by role. The developer who changes the harness attaches the eval results artifact to the pull request as part of its evidence bundle. The tech lead owns evals/, evals/baseline-rate and .github/workflows/agent-evals.yml through CODEOWNERS, with a required code-owner review in branch protection or a ruleset, reviews the per-case results rather than the prose of the new rules, and is the only person who moves the baseline. That split keeps the agent, and the person proposing the change, from editing the checks that judge it.

What breaks in continuous evals, and how to recover

Section titled “What breaks in continuous evals, and how to recover”
SymptomCauseRecovery
The canary fails, and results look like the harness change had no effectThe harness was not loaded: --bare in the Claude Code call, or an untrusted worktree in CodexRemove --bare; add the trust_level entry; the canary must fail before the fix and pass after
An old rule still affects resultsA nested CLAUDE.md or AGENTS.md in a subdirectory of the base commit survived the overlayExtend the overlay list in run.sh to the nested files, or delete them before copying
The agent’s changes pass a check they should failThe grader’s tests were visible in the worktree, so the agent wrote to the testKeep hidden tests in the case directory; run.sh deletes evals/ from the worktree
The pass rate swings between identical runsOne run per case, or too few casesThree runs or more; decide on cases that fail in most runs; grow the suite
The job blocks merges with no harness changeThe Monday run met a model or service changeRerun once; if it holds, open an issue, compare with the new-model playbook, and fix the harness or accept a new baseline in a reviewed pull request
One case burns the budgetA loop of failing tool calls--max-budget-usd per run, timeout 900 per case, timeout-minutes on the job
Fork pull requests show a skipped eval jobForks get no secrets, and the job guard skips them by designHave a maintainer re-create the change on a branch in the repository
Scope checks fail on files the agent never touchedThe harness overlay appeared in the diffrun.sh commits the overlay before the agent runs; keep that step

Where to go next with harness evals

Section titled “Where to go next with harness evals”

Before this page: the harness you are testing, in concise repository context, agent skills and hooks as guardrails. After it: agent-driven browser verification, the next Test-stage question.

Evaluating agents and CLAUDE.md changes A/B test one rules change with promptfoo, Inspect SWE or Harbor.
The Test stage The session feedback loop that continuous evals protect.
Model-graded checks Calibrate a judge before it can block a merge.
The harness Every layer the suite regression-tests, and who owns it.
Developer answer key Every scorecard question mapped to its canonical page.

Frequently asked questions

When should continuous evals run for an AI coding harness?

Continuous evals run real repository tasks headless in CI on every change to CLAUDE.md, AGENTS.md, rules, skills, hooks, the pinned CLI version, or the model, as well as on a weekly schedule on main.

How do continuous evals prevent agent configuration regressions?

Deterministic checks grade each headless task run, block merges when repeated pass rates drop below the baseline, and grow stronger whenever a real incident is converted into a new eval task.

All 577 guides Free resources: Compare tools Glossary By role

Written by

See what's new on the platform.

Privacy Policy Terms of Service Cookie settings llms.txt pricing.md RSS