Continuous Evals — Regression-Testing Agent Harnesses

Continuous evals are automated regression suites that measure the performance, compliance, and accuracy of your AI agent harness (CLAUDE.md, .cursor/rules, .agents/skills, and hooks) in CI. When prompts, instruction files, tools, or models change, representative tasks detect material regressions and produce evidence for a human decision; no finite suite guarantees the absence of degradation.

Scorecard question: How do you regression-test your agent configuration (CLAUDE.md, rules, skills, hooks)? Max‑score answer (3 pts): Continuous CI eval workflow (.github/workflows/agent-evals.yml) running on config/skill PRs, with production incidents added as regression tests.

Why this matters in 2026

Agent instruction files and skills steer millions of lines of generated code, yet most teams treat them like loose documentation. A well-intentioned developer tweaks a rule or upgrades a model default, only to discover days later that the agent stopped following brand guidelines or began ignoring security policies across the entire engineering org.

In the AI-native SDLC, agent configuration is code and must be tested like code. By maintaining an evaluation suite of 20–50 representative tasks with deterministic pass/fail checks, teams run headless evals on every PR that touches .claude/, .cursor/, or .agents/. Furthermore, every production incident is transformed into a new eval case, permanently preventing the recurrence of that failure mode.

What “max score” actually looks like

A max-score Q17 setup demonstrates four operational capabilities:

Dedicated CI workflow: An automated workflow (e.g., .github/workflows/agent-evals.yml) triggers on any change to agent rules, instructions, or skills.
Deterministic grading: Each eval case tests an agent run against concrete criteria: unit tests pass, linter exits 0, no prohibited APIs are introduced, and required patterns exist.
Incident-to-eval pipeline: Post-mortem actions require writing an eval task reproducing the incident before the incident ticket can be closed.
Pass-rate gating: Changes to rules or skills that drop the benchmark pass rate are blocked from merging to main.

Step-by-step implementation

Curate an initial task benchmark.

Collect 15–30 historical tasks representing common workflows in your repository:
- Adding a new API endpoint with authentication
- Refactoring a database query to use parameterized inputs
- Writing a React component following design tokens
- Handling an edge-case error in an asynchronous worker

Structure eval test cases.

Create an evals/ directory in your repository. For each case, store the starting prompt, expected file changes, and verification command:

{
  "id": "eval-auth-middleware",
  "description": "Ensure agent adds auth check to unprotected routes",
  "prompt": "Create a new route /api/v1/user/settings in src/routes/settings.ts following our API standards.",
  "checks": [
    "test -f src/routes/settings.ts",
    "grep -q 'requireAuth' src/routes/settings.ts",
    "npm run typecheck",
    "npm test tests/routes/settings.test.ts"
  ]
}

Build the automated CI workflow.

Create .github/workflows/agent-evals.yml to run the suite headless on PRs modifying agent files:

name: Agent Evals
on:
  pull_request:
    paths:
      - '.claude/**'
      - '.cursor/**'
      - '.agents/**'
      - 'CLAUDE.md'
      - 'AGENTS.md'

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Run Headless Agent Evals
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          npm ci
          # Run eval runner script using headless agent mode
          npx tsx scripts/run-evals.ts

Execute headless agent runs in the runner.

Use your primary tool’s non-interactive execution mode:
- Claude Code: claude -p "PROMPT" inside an isolated runner.
- Codex: openai/codex-action@v1 or codex exec.
- Cursor: Cursor Cloud Agent automation or CLI in a sandboxed container.
Convert every incident into a regression eval.

Whenever an incident or severe defect escapes to production, document the prompt and failure mode as a new JSON file in evals/. Commit the eval alongside the incident post-mortem.

Common pitfalls

Testing models instead of harnesses: Writing evals that measure raw LLM benchmark trivia rather than testing your specific repository conventions and skills.
Flaky assertions: Relying on LLM judges for grading when deterministic bash checks (npm test, grep, linter) provide faster and more reliable verdicts.
Running evals only manually: Relying on developers to remember to run evals locally before pushing rule changes. Enforce the gate in CI.

How to verify you’re there

Modifying CLAUDE.md or .cursor/rules/ triggers the Agent Evals workflow in CI.
The eval suite pass rate is tracked over time and visible on a dashboard.
Every production incident post-mortem includes a committed eval file.
Rule changes that cause agent hallucinations or regressions fail CI and are blocked from merging.