Skip to content

@wondelai/harness-audit evaluates a repository deterministically across six weighted Software Factory stations—Intent, Harness, Loop, Graph, Verification, and Release—and maps the resulting evidence onto Dan Shapiro’s Autonomy Ladder (Level 0–5). Every check runs locally with zero network uploads and links directly to its corresponding pattern in this canon.

Developers, tech leads, and CTOs use this reference to inspect why a repository received a specific score or Autonomy Level cap, which concrete artifacts unblock the next rung, and how to turn audit checks into continuous CI guardrails. For step-by-step usage, optional interviews, and remediation prompts, see the harness audit guide.

Evaluates whether the repository provides every coding agent concise, non-duplicated context and specification discipline before code edits begin.

Check areaWhat passesGuide
Root instruction fileAGENTS.md (or tool equivalent) exists with real repo boundaries and verification commandsAGENTS.md and concise context
Single source of truthCLAUDE.md delegates via @AGENTS.md rather than duplicating policy textShared team rules
Instruction budgetRoot brief stays compact (≤ 150 lines) without generic filler or copied boilerplatePruning context files
Scoped rules & skillsPath-scoped rules (.claude/rules/, .cursor/rules/*.mdc) and reusable SKILL.md skillsAgent skills
Specification artifactsPR/issue templates or spec and plan structure capturing acceptance criteria before codingSpec-driven development

Evaluates whether permissions, lifecycle hooks, MCP integrations, and secret boundaries are enforced mechanically by the environment rather than by prompt politeness.

Check areaWhat passesGuide
Permissions & sandboxExplicit allow/deny or sandbox/approval posture (flags retired policies such as Codex untrusted)Permissions and sandboxing
Lifecycle hooksDeterministic PreToolUse, PostToolUse, or Stop hooks with loop protection (stop_hook_active)Agent hooks
MCP hygieneZero archived or hallucinated packages, deprecated /sse endpoints, or plaintext Bearer tokensModel Context Protocol
Secret protection.env / .dev.vars gitignored, .env.example committed, zero leaked credentials in configAgent identity and secrets

Evaluates whether an agent can verify its own edits fast and reproducibly through one deterministic command.

Check areaWhat passesGuide
Single-command verifyUnified check / verify / test + lint + typecheck command executable locally and in CIMaking a codebase agent-ready
Strict static typesStrict TypeScript or equivalent static type checking enabled across source pathsCode quality and verification
Fast linter & formatterDeterministic formatter and linter configured with non-interactive CI check modeThe feedback loop
ReproducibilityCommitted lockfile and pinned runtime version matching CIAgent-ready codebase

Evaluates whether concurrent agents can execute parallel tasks without colliding on branches, ports, databases, or working state.

Check areaWhat passesGuide
Worktree isolationWorktree workflow script or .worktreeinclude setup isolating checkouts per taskGit worktrees
Ephemeral ports & statePer-worktree local ports and isolated local database/cache stateEphemeral environments
Subagents & workflowsScoped read-only or worktree-isolated subagent definitions (.claude/agents/, Codex [agents])Orchestration patterns

Evaluates whether automated oracles are strong enough to justify reviewing evidence instead of every generated line.

Check areaWhat passesGuide
Unit & E2E suitesSubstantive unit and end-to-end tests covering behavioral contractsOracle strength
Oracle protectionCODEOWNERS or CI rules guarding test suites, gates, and workflows against silent weakeningProtecting the oracle
Test quality scanLow .skip/.only counts, zero tautological assertions, minimal lint or type suppressionsSlop detection
Automated PR reviewConfigured Bugbot (.cursor/BUGBOT.md), Claude/Codex review, or automated PR review gatesReviewing agent PRs
Fitness functionsAutomated architectural layering, bundle, dependency, or security checksArchitecture fitness functions

Evaluates whether merged agent work reaches production safely with auditable evidence and rollback readiness.

Check areaWhat passesGuide
CI pull-request gatesRequired automated lint, typecheck, and test workflows running on every PRDeploy and production gates
Evidence PR templatePR checklist capturing executed checks, risk class, sensitive paths, and rollback notesThe evidence bundle
Dependency & observabilityAutomated dependency verification plus error tracking and structured production telemetryKeeping an agent codebase healthy

How station scores and hard level caps prevent inflated ratings

Section titled “How station scores and hard level caps prevent inflated ratings”

Weighted points summarize breadth, while hard level caps enforce architectural prerequisites. A repository with a tidy AGENTS.md and rich skills still caps at Level 2 when it lacks deterministic test suites and required CI checks:

Autonomy LevelPrerequisite station evidence
Level 1–2Root instruction file plus reproducible build and test commands
Level 3Single-source instructions, fast local verification loop, explicit permission posture, and CI PR gates
Level 4Strong Verification station (real tests, oracle protection, automated PR review) plus Graph isolation
Level 5Complete six-station coverage including evidence bundles, fitness functions, and release observability

How to verify your repository and enforce a CI bar

Section titled “How to verify your repository and enforce a CI bar”

Run the scanner locally to inspect every failing check, and wire CI mode into pull requests so harness health ratchets upward over time:

Terminal window
npx @wondelai/harness-audit
npx @wondelai/harness-audit --ci --min-score 65

Pass --deep when you want qualitative inspection of instruction signal-to-noise and test oracle strength using your local claude or codex CLI; zero code leaves your machine during the default deterministic scan. For tool-specific configuration schemas flagged by any check, consult the official docs for Claude Code, OpenAI Codex, Cursor, Agent Skills, and MCP.

When audit checks flag false positives or stubborn caps

Section titled “When audit checks flag false positives or stubborn caps”
  • A monorepo scores lower than individual packages. Running the scanner at a wrapper root that lacks top-level scripts misses package-level test suites. Recovery: run the audit against each active service root or expose unified root verification scripts that fan out cleanly.
  • Retired policy keys fail Station 2 even though local runs appear to work. Legacy config such as Codex approval_policy = "untrusted" or deprecated .cursorrules files breaks silently or fails on upgraded runners. Recovery: migrate to active sandbox_mode + approval_policy (on-request, never, granular) or .cursor/rules/*.mdc per official docs.
  • High test counts still trigger Station 5 warnings. Suites padded with tautological assertions, .skip markers, or broad type suppressions pass line counts without proving behavior. Recovery: audit failing assertions with oracle strength and lock test paths in CODEOWNERS.
  • Checks pass on one laptop but fail in CI. User-scoped settings (~/.claude/, ~/.codex/) never travel to CI runners or cloud VMs. Recovery: commit shared permissions, hooks, rules, and .mcp.json into the repository.

Did this guide help you?

One tap, no sign-in.

not at all a lot