Harness audit checks reference
PL@wondelai/harness-audit evaluates a repository deterministically across six weighted Software Factory stations—Intent, Harness, Loop, Graph, Verification, and Release—and maps the resulting evidence onto Dan Shapiro’s Autonomy Ladder (Level 0–5). Every check runs locally with zero network uploads and links directly to its corresponding pattern in this canon.
Developers, tech leads, and CTOs use this reference to inspect why a repository received a specific score or Autonomy Level cap, which concrete artifacts unblock the next rung, and how to turn audit checks into continuous CI guardrails. For step-by-step usage, optional interviews, and remediation prompts, see the harness audit guide.
Station 1 — Intent
Section titled “Station 1 — Intent”Evaluates whether the repository provides every coding agent concise, non-duplicated context and specification discipline before code edits begin.
| Check area | What passes | Guide |
|---|---|---|
| Root instruction file | AGENTS.md (or tool equivalent) exists with real repo boundaries and verification commands | AGENTS.md and concise context |
| Single source of truth | CLAUDE.md delegates via @AGENTS.md rather than duplicating policy text | Shared team rules |
| Instruction budget | Root brief stays compact (≤ 150 lines) without generic filler or copied boilerplate | Pruning context files |
| Scoped rules & skills | Path-scoped rules (.claude/rules/, .cursor/rules/*.mdc) and reusable SKILL.md skills | Agent skills |
| Specification artifacts | PR/issue templates or spec and plan structure capturing acceptance criteria before coding | Spec-driven development |
Station 2 — Harness
Section titled “Station 2 — Harness”Evaluates whether permissions, lifecycle hooks, MCP integrations, and secret boundaries are enforced mechanically by the environment rather than by prompt politeness.
| Check area | What passes | Guide |
|---|---|---|
| Permissions & sandbox | Explicit allow/deny or sandbox/approval posture (flags retired policies such as Codex untrusted) | Permissions and sandboxing |
| Lifecycle hooks | Deterministic PreToolUse, PostToolUse, or Stop hooks with loop protection (stop_hook_active) | Agent hooks |
| MCP hygiene | Zero archived or hallucinated packages, deprecated /sse endpoints, or plaintext Bearer tokens | Model Context Protocol |
| Secret protection | .env / .dev.vars gitignored, .env.example committed, zero leaked credentials in config | Agent identity and secrets |
Station 3 — Loop
Section titled “Station 3 — Loop”Evaluates whether an agent can verify its own edits fast and reproducibly through one deterministic command.
| Check area | What passes | Guide |
|---|---|---|
| Single-command verify | Unified check / verify / test + lint + typecheck command executable locally and in CI | Making a codebase agent-ready |
| Strict static types | Strict TypeScript or equivalent static type checking enabled across source paths | Code quality and verification |
| Fast linter & formatter | Deterministic formatter and linter configured with non-interactive CI check mode | The feedback loop |
| Reproducibility | Committed lockfile and pinned runtime version matching CI | Agent-ready codebase |
Station 4 — Graph
Section titled “Station 4 — Graph”Evaluates whether concurrent agents can execute parallel tasks without colliding on branches, ports, databases, or working state.
| Check area | What passes | Guide |
|---|---|---|
| Worktree isolation | Worktree workflow script or .worktreeinclude setup isolating checkouts per task | Git worktrees |
| Ephemeral ports & state | Per-worktree local ports and isolated local database/cache state | Ephemeral environments |
| Subagents & workflows | Scoped read-only or worktree-isolated subagent definitions (.claude/agents/, Codex [agents]) | Orchestration patterns |
Station 5 — Verification
Section titled “Station 5 — Verification”Evaluates whether automated oracles are strong enough to justify reviewing evidence instead of every generated line.
| Check area | What passes | Guide |
|---|---|---|
| Unit & E2E suites | Substantive unit and end-to-end tests covering behavioral contracts | Oracle strength |
| Oracle protection | CODEOWNERS or CI rules guarding test suites, gates, and workflows against silent weakening | Protecting the oracle |
| Test quality scan | Low .skip/.only counts, zero tautological assertions, minimal lint or type suppressions | Slop detection |
| Automated PR review | Configured Bugbot (.cursor/BUGBOT.md), Claude/Codex review, or automated PR review gates | Reviewing agent PRs |
| Fitness functions | Automated architectural layering, bundle, dependency, or security checks | Architecture fitness functions |
Station 6 — Release
Section titled “Station 6 — Release”Evaluates whether merged agent work reaches production safely with auditable evidence and rollback readiness.
| Check area | What passes | Guide |
|---|---|---|
| CI pull-request gates | Required automated lint, typecheck, and test workflows running on every PR | Deploy and production gates |
| Evidence PR template | PR checklist capturing executed checks, risk class, sensitive paths, and rollback notes | The evidence bundle |
| Dependency & observability | Automated dependency verification plus error tracking and structured production telemetry | Keeping an agent codebase healthy |
How station scores and hard level caps prevent inflated ratings
Section titled “How station scores and hard level caps prevent inflated ratings”Weighted points summarize breadth, while hard level caps enforce architectural prerequisites. A repository with a tidy AGENTS.md and rich skills still caps at Level 2 when it lacks deterministic test suites and required CI checks:
| Autonomy Level | Prerequisite station evidence |
|---|---|
| Level 1–2 | Root instruction file plus reproducible build and test commands |
| Level 3 | Single-source instructions, fast local verification loop, explicit permission posture, and CI PR gates |
| Level 4 | Strong Verification station (real tests, oracle protection, automated PR review) plus Graph isolation |
| Level 5 | Complete six-station coverage including evidence bundles, fitness functions, and release observability |
How to verify your repository and enforce a CI bar
Section titled “How to verify your repository and enforce a CI bar”Run the scanner locally to inspect every failing check, and wire CI mode into pull requests so harness health ratchets upward over time:
npx @wondelai/harness-auditnpx @wondelai/harness-audit --ci --min-score 65Pass --deep when you want qualitative inspection of instruction signal-to-noise and test oracle strength using your local claude or codex CLI; zero code leaves your machine during the default deterministic scan. For tool-specific configuration schemas flagged by any check, consult the official docs for Claude Code, OpenAI Codex, Cursor, Agent Skills, and MCP.
When audit checks flag false positives or stubborn caps
Section titled “When audit checks flag false positives or stubborn caps”- A monorepo scores lower than individual packages. Running the scanner at a wrapper root that lacks top-level scripts misses package-level test suites. Recovery: run the audit against each active service root or expose unified root verification scripts that fan out cleanly.
- Retired policy keys fail Station 2 even though local runs appear to work. Legacy config such as Codex
approval_policy = "untrusted"or deprecated.cursorrulesfiles breaks silently or fails on upgraded runners. Recovery: migrate to activesandbox_mode+approval_policy(on-request,never,granular) or.cursor/rules/*.mdcper official docs. - High test counts still trigger Station 5 warnings. Suites padded with tautological assertions,
.skipmarkers, or broad type suppressions pass line counts without proving behavior. Recovery: audit failing assertions with oracle strength and lock test paths inCODEOWNERS. - Checks pass on one laptop but fail in CI. User-scoped settings (
~/.claude/,~/.codex/) never travel to CI runners or cloud VMs. Recovery: commit shared permissions, hooks, rules, and.mcp.jsoninto the repository.
Next steps from the check catalogue
Section titled “Next steps from the check catalogue”Did this guide help you?
One tap, no sign-in.
What will you do with it?
How did it read?
What got in the way? Pick any that apply
Your role?
Thanks. I read every answer. Michał Jaskólski
Did this guide help?
One tap each. Close whenever you like.
no sign-in