Auditing your agent harness with @wondelai/harness-audit
PL@wondelai/harness-audit is an open-source CLI that scores any repository from Level 0 to Level 5 across the six Software Factory stations—Intent, Harness, Loop, Graph, Verification, and Release—by inspecting committed files, CI gates, test oracles, and agent configurations locally in under two seconds. Developers, tech leads, and CTOs use it to replace subjective self-assessment with verifiable repository evidence.
Most engineering teams overestimate their autonomy level because questionnaires measure aspirations rather than what actually sits in Git. A team lead may believe strict guardrails are live while CLAUDE.md has bloated past 400 lines, .codex/config.toml still specifies the retired untrusted approval policy, and CI runs zero protection against agents weakening test files.
What an automated harness audit measures
Section titled “What an automated harness audit measures”Subjective forms record how a team hopes it works. @wondelai/harness-audit reads the exact environment Claude Code, Codex, and Cursor encounter when opening a session, mapping every finding onto Dan Shapiro’s Autonomy Ladder (January 2026) and the six Software Factory stations:
| Station | What deterministic repository checks verify | Core canon guide |
|---|---|---|
| 1. Intent | Concise AGENTS.md (≤ 150 lines), thin @AGENTS.md delegation in CLAUDE.md, scoped rules, skills, and spec/PR templates | AGENTS.md & context |
| 2. Harness | Explicit permission/sandbox posture (flagging retired Codex untrusted), lifecycle hooks with stop_hook_active, MCP hygiene, and secret protection | Permissions & sandboxes · Agent hooks |
| 3. Loop | Unified single-command verification (lint, typecheck, test, build), strict static typing, deterministic formatter, committed lockfile | Agent-ready codebase |
| 4. Graph | Worktree workflow scripts or .worktreeinclude, isolated local ports/state, and scoped subagent definitions | Git worktrees · Ephemeral environments |
| 5. Verification | Real unit and E2E suites, CODEOWNERS/CI oracle protection, low .skip/suppression slop, automated PR review (e.g. .cursor/BUGBOT.md), fitness functions | Oracle strength · Protecting the oracle |
| 6. Release | PR CI gates, evidence-bundle PR template, automated dependency checks, and production error/telemetry observability | Evidence bundle · Codebase health |
For the complete per-check inventory, see the harness audit checks reference.
Why hard level caps prevent inflated Level 4 scores
Section titled “Why hard level caps prevent inflated Level 4 scores”A repository with a spotless AGENTS.md and twenty Agent Skills—yet no test suite, no oracle protection, and weak CI gates—still operates at Level 2. Averaging raw points hides structural weaknesses, so @wondelai/harness-audit applies non-negotiable level caps:
| Claimed Level | Required minimum evidence in the repository |
|---|---|
| Level 1–2 | Baseline instruction file and reproducible build/test commands |
| Level 3 | Single-source instructions, deterministic verification loop, explicit permission posture, and CI gates |
| Level 4 | Strong Verification station (real tests, oracle protection, automated PR review) plus Graph isolation |
| Level 5 | Full six-station coverage including evidence bundles, architectural fitness functions, and release observability |
Level caps make the audit useful for CTOs and platform leads comparing dozens of repositories: a high Intent score cannot disguise an unguarded test oracle or missing sandbox boundary.
Run the audit and close gaps step by step
Section titled “Run the audit and close gaps step by step”-
Run the zero-config local scan. Execute from any repository root—zero API keys, zero network calls, zero code uploaded:
Terminal window npx @wondelai/harness-audit -
Capture workflow signals files cannot show. Add the optional 7-question terminal interview for team adoption, PR review queue age, and cost visibility:
Terminal window npx @wondelai/harness-audit --interview --role lead -
Generate a tailored fix prompt or minimal baseline. Export a repository-specific remediation prompt for Claude Code, Codex, or Cursor—or preview a clean non-destructive scaffold before writing anything:
Terminal window npx @wondelai/harness-audit --prompt --out HARNESS-FIX-PROMPT.mdnpx @wondelai/harness-audit init --dry-run -
Share reports and enforce a regression floor in CI. Export a standalone HTML report for leadership reviews and add a CI gate so harness scores never silently decay on future PRs:
Terminal window npx @wondelai/harness-audit --format html --out harness-report.htmlnpx @wondelai/harness-audit --ci --min-score 65
Verify that audit remediation actually improved safety
Section titled “Verify that audit remediation actually improved safety”Raising a score only counts when the underlying controls catch real defects. Pair every audit-driven pull request with three verification checks:
- Re-run the deterministic scan in CI: Confirm
--ci --min-scorepasses without weakening thresholds and that no level cap remains active on the station you fixed. - Run live canary probes: Adding a hook file or sandbox block satisfies static detection; prove refusal with hook fixtures, a sandbox canary, and a codebase defect canary.
- Track multi-repo baselines over time: Store periodic HTML or JSON outputs across core repositories so engineering leadership tracks real structural progress rather than raw AI code volume (metrics frameworks).
When harness audits mislead teams
Section titled “When harness audits mislead teams”- Teams game the score with empty boilerplate. Adding placeholder
AGENTS.mdfiles, dummy hooks, or tautological tests ticks static checkboxes without improving safety. Pair static scans with--deepinspection, slop detection, and canary tests. - Engineers chase points in strong stations while ignoring level caps. Polishing Intent from 85 to 95 does not raise an organization capped at Level 2 by weak Verification. Fix the lowest-scoring gating station first.
- Running
initwithout auditing existing configs. Always preview withnpx @wondelai/harness-audit init --dry-runfirst so generated scaffolds align with your team’s existingAGENTS.md, package manager, and CI layout. - Treating one audit as a permanent certification. Tool conventions, model capabilities, and CI configs drift every quarter. Keep
--ci --min-scorein pull-request workflows and re-evaluate your threshold after every major upgrade.
Interactive guide, books, and team support
Section titled “Interactive guide, books, and team support”Explore the interactive breakdown at /en/harness-audit/, browse the open canon editions at /en/books/, or consult official vendor documentation for tool-specific remediation syntax: Claude Code docs, OpenAI Codex docs, Cursor docs, Agent Skills, and MCP.
Did this guide help you?
One tap, no sign-in.
What will you do with it?
How did it read?
What got in the way? Pick any that apply
Your role?
Thanks. I read every answer. Michał Jaskólski
Did this guide help?
One tap each. Close whenever you like.
no sign-in