Skip to content
AI Developer Toolkit

Find your level — free Buy the Toolkit Buy Toolkit Buy the Toolkit Buy Toolkit Open the guides Read
Navigation menu
Buy the Toolkit Find your level — free Buy the Toolkit Open the guides Sign in

Evidence-based rating and iteration

PL

Level 4 How-to Updated

An evidence-based rating loop scores a handoff artifact, such as a visual direction, prototype, design contract, or spec, against written acceptance criteria. A reviewer that did not produce the artifact rates it, every point cites reproducible evidence, the lowest dimension is the handoff score, and the loop ends at an explicit threshold or a named stop condition.

The agent revised the checkout design contract four times and reported 7/10, then 8/10, 8.5/10, and finally 9.5/10. You open version four and the declined-card state is still missing, a new colour has appeared, and nobody can say which of the earlier findings were fixed or quietly dropped. This page is for developers and tech leads who hand plan-stage artifacts between people and agents, and want a score they can gate on instead of a number the author gave itself.

What you get from an evidence-based rating loop

Section titled “What you get from an evidence-based rating loop”
  • A four-dimension rubric where the minimum wins, and a JSON Schema that forces every rating into the same shape.
  • A read-only review command for Claude Code and Codex, and the equivalent routine in Cursor.
  • A jq gate that computes the score itself instead of trusting the reviewer’s arithmetic.
  • A findings ledger with a disposition for every finding, so fixes are tracked rather than remembered.
  • Three copy-paste prompts: rate, fix, and re-rate.
  • Stop conditions, a calibration routine for the reviewer, and recovery steps for the ways the loop goes wrong.

When is a rating loop worth running?

Section titled “When is a rating loop worth running?”

The loop costs a review run per iteration plus a human decision at the end. Use it where a wrong handoff means rework downstream, not on every artifact.

SituationUse
An artifact in the four-artifact pipeline came back from review more than onceFull loop
A design contract or spec.md that several implementation tasks will readFull loop, threshold 9
A customer-facing flow with accessibility or legal exposureFull loop plus a named human acceptance for those dimensions
An internal screen built only from existing componentsOne rating pass, no iteration unless a blocker appears
A copy or spacing fixNo loop; normal review of the change

The loop rates artifacts that humans accept. It assumes the criteria already exist; if they do not, write them first with executable acceptance criteria. Rating the code the artifact produces is a different job, covered by model-graded checks and agent PR review.

How the rubric scores a handoff

Section titled “How the rubric scores a handoff”

Four dimensions cover what goes wrong in plan-stage handoffs. Each scores 0 to 10, and the handoff score is the minimum, not the average: a perfect visual system does not compensate for a missing error state.

DimensionScores 10 whenTypical evidence
Intent fidelityEvery acceptance criterion and non-goal in intent.md or spec.md is met, each cited by IDCriterion ID, the section or frame that satisfies it
State completenessEvery required state, breakpoint, content variant, keyboard path, and failure is presentRow in states.md, fixture name, screenshot path
System coherenceOnly canonical tokens, components, interaction patterns, and terms are usedToken gate output, component list, glossary term
Evidence qualityAnother reviewer can reproduce each finding from the route, state, viewport, and check namedThe command or route to rerun, artifact version

A 10 does not mean “looks excellent”. It means every declared criterion has reproducible proof and no blocker is open. For normal work, set the ship threshold at 9 with no open blocker, no criterion left fail or not-tested, and a human acceptance recorded.

Save this schema as design/rating.schema.json. It makes the reviewer return the same structure every time, so a script can gate on it. Every object sets additionalProperties: false and lists all its properties as required, because Codex’s --output-schema expects a strict schema. Keep the 0 to 10 range in the prompt, and let the jq gate in step 4 reject any score outside it, so the range is enforced by the script rather than trusted.

{
  "type": "object",
  "additionalProperties": false,
  "required": ["artifact_version", "criteria", "dimensions", "findings"],
  "properties": {
    "artifact_version": { "type": "string" },
    "criteria": {
      "type": "array",
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["id", "result", "evidence"],
        "properties": {
          "id": { "type": "string" },
          "result": { "type": "string", "enum": ["pass", "fail", "not-tested"] },
          "evidence": { "type": "string" }
        }
      }
    },
    "dimensions": {
      "type": "object",
      "additionalProperties": false,
      "required": ["intent", "states", "system", "evidence"],
      "properties": {
        "intent": { "type": "integer" },
        "states": { "type": "integer" },
        "system": { "type": "integer" },
        "evidence": { "type": "integer" }
      }
    },
    "findings": {
      "type": "array",
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["id", "criterion", "severity", "evidence", "fix"],
        "properties": {
          "id": { "type": "string" },
          "criterion": { "type": "string" },
          "severity": { "type": "string", "enum": ["blocker", "major", "minor"] },
          "evidence": { "type": "string" },
          "fix": { "type": "string" }
        }
      }
    }
  }
}

Run the rating loop

Section titled “Run the rating loop”

  1. Freeze the version you rate.

    Commit the artifact, or pin the design-tool link to a version, and record that identifier in design/checkout/VERSION. A rating that cannot name the exact version it scored cannot be compared with the next one. Keep all ratings for one artifact in design/<feature>/ratings/, numbered r1.json, r2.json, and so on. Increment the round number on every run and never write over an existing rating file: the re-rate compares against the previous one.

  2. Run the deterministic checks first.

    Anything a machine can decide, a machine decides: the Playwright state spec, the axe scan, and the token gate from the four-artifact pipeline, or a spec linter for text artifacts. Put the commands in design/checkout/checks/run.sh so every round, and the fixer, runs exactly the same checks, and have it write its output to design/checkout/checks/. The reviewer cites that output as evidence; a failing check caps its dimension, whatever the reviewer thinks of the rest.

  3. Rate the artifact in a fresh, read-only context.

    The reviewer must not be the session that produced the artifact, and it must not be able to edit it. Save the prompt below as design/checkout/rate-prompt.md.

    Run it headless so the output lands in a file:

    Terminal window
    claude -p --permission-mode dontAsk \
      --allowedTools "Read Grep Glob" \
      --disallowedTools "Edit Write NotebookEdit Bash" \
      --output-format json \
      --json-schema "$(cat design/rating.schema.json)" \
      "$(cat design/checkout/rate-prompt.md)" \
      | jq '.structured_output' > design/checkout/ratings/r1.json

    dontAsk denies every tool call that would prompt (reads in the working directories and read-only commands still run) and --disallowedTools removes the write tools even if a settings file allows them, so the reviewer can read but not write; with --json-schema, the validated object arrives in the structured_output field (checked on Claude Code 2.1.283).

    In Claude Code, a read-only custom subagent gives the same separation for interactive work inside a session.

  4. Gate on the score with a script, not on the reviewer’s summary.

    The reviewer supplies judgements; the script does the arithmetic. This gate exits non-zero unless the rating names the frozen version, every dimension is between 0 and 10, the minimum dimension is at least 9, no blocker is open, and every criterion passes. Run it on the newest rating file:

    Terminal window
    jq -e --arg v "$(cat design/checkout/VERSION)" '
      {version_ok: (.artifact_version == $v),
       in_range: ([.dimensions[]] | all(. >= 0 and . <= 10)),
       score: ([.dimensions[]] | min),
       blockers: ([.findings[] | select(.severity == "blocker")] | length),
       not_passing: ([.criteria[] | select(.result != "pass")] | length)}
      | ., (.version_ok and .in_range and .score >= 9 and .blockers == 0 and .not_passing == 0)
    ' design/checkout/ratings/r1.json

    It prints the five values and then true or false; -e sets the exit code from that last value, so the same command works in a terminal, a pre-merge job, or a loop script. A rating of the wrong version, or one with a score outside 0 to 10, fails the gate instead of passing quietly.

  5. Give every finding a disposition in the ledger.

    Copy the findings into design/checkout/findings.md, one row each, and have the artifact owner set the disposition. Findings are never deleted; they change status.

    | ID  | Round | Criterion | Severity | Status   | Owner        | Note                          |
    | F-1 | r1    | AC-4      | blocker  | fixed    | design lead  | fixed in r2, verified r3      |
    | F-2 | r1    | AC-7      | major    | accepted | product      | ships without offline state   |
    | F-3 | r2    | AC-4      | minor    | open     | design lead  |                               |

    accepted means a named human chose to ship with the gap; duplicate points at another ID; fixed is set by the artifact owner from fix-notes.md, and stays provisional until the next re-rate confirms it with evidence; open is everything else.

  6. Fix in a separate task.

    The fixer gets the ledger, not the whole conversation with the reviewer. It may change the artifact, and nothing else.

  7. Re-rate in a new context, with the ledger.

    A fresh reviewer rates the new version and also checks the previous ledger, so old findings are verified as well as new ones found. Save the prompt below as design/checkout/rerate-prompt.md.

    Run it like step 3, with two changes. The new rating is written to a temporary file and moves into ratings/ only once it is valid JSON, and the previous round is named on the prompt’s last line. A plain > r2.json would not be safe here: the shell creates the empty target before the reviewer starts, so a reviewer told to read “the newest file” would compare against an empty r2.json and find no regressions. For the first re-rate, prev is r1.json and next is r2.json:

    Terminal window
    prev=design/checkout/ratings/r1.json
    next=design/checkout/ratings/r2.json
    tmp=$(mktemp)
    test -s "$prev" && test ! -e "$next" \
      && claude -p --permission-mode dontAsk \
           --allowedTools "Read Grep Glob" \
           --disallowedTools "Edit Write NotebookEdit Bash" \
           --output-format json \
           --json-schema "$(cat design/rating.schema.json)" \
           "$(cat design/checkout/rerate-prompt.md)
    Previous rating file: $prev" \
         | jq -e '.structured_output' > "$tmp" \
      && mv "$tmp" "$next"

    The chain fails closed and changes nothing in ratings/ if the previous file is missing or empty, if the next file already exists, or if the reviewer exits with an error or returns no rating (jq -e exits non-zero on empty or null output). The next round sets prev to r2.json and next to r3.json, and so on. Then run the step 4 gate on the new file and update the ledger.

  8. Stop, and hand the decision to a human.

    When the gate passes, or a stop condition below applies, the loop ends. A named person accepts the artifact and the ledger, and both travel with it into spec.md, plan.md, and the pull request’s evidence bundle.

Let the reviewer collect browser evidence

Section titled “Let the reviewer collect browser evidence”

A reviewer that reads files can only cite the check output you saved. To let it open the running app and collect state and viewport evidence itself, load the Playwright MCP server (@playwright/mcp, 0.0.82 on npm on 2026-09-26) for the review run only. Pin that version so every round collects evidence with the same browser tooling.

Replace the --allowedTools line of the step 3 command with these three options. --strict-mcp-config loads only this server and ignores the MCP servers in your user and project configuration, some of which may write; mcp__playwright allows every tool of that one server.

Terminal window
--strict-mcp-config \
--mcp-config '{"mcpServers":{"playwright":{"command":"npx","args":["@playwright/mcp@0.0.82"]}}}' \
--allowedTools "Read Grep Glob mcp__playwright"

When should the rating loop stop?

Section titled “When should the rating loop stop?”

Stop iterating and hand control to a human when any of these holds:

  • The gate passes and every remaining finding has a disposition.
  • Two rounds in a row do not improve the same failing criterion. More rounds rarely help; the criterion, the artifact, or the tool is wrong.
  • A product, brand, legal, privacy, or accessibility decision is missing. An agent cannot make it, however many times it is asked.
  • Fixing a finding would change accepted intent, a non-goal, or the architecture. That is a new decision, not an iteration.
  • The reviewer cannot reproduce the artifact or collect the evidence a criterion needs.
  • You reach your round cap. Set one before starting; three rounds is a reasonable default for one artifact.

Who signs off, and how do you trust the reviewer?

Section titled “Who signs off, and how do you trust the reviewer?”

The rating is evidence for a human decision, never the decision itself. The agent that produced the artifact can critique its own work as a first pass, but it cannot certify it, because it can miss the same defect it created.

DecisionMade byBased on
Criteria and thresholdArtifact owner, before round 1acceptance.md, risk of the change
Each finding’s dispositionArtifact owner; product for scope, design-system owner for tokensLedger evidence
accepted for an accessibility, legal, or privacy gapThe named owner of that areaFinding evidence and the reason recorded
Handoff acceptanceArtifact ownerPassing gate, ledger, last rating file

Check the reviewer as you would check any test. Two routines keep it honest:

  • Seed a known defect. Before relying on a new rubric or prompt, remove one required state from a copy of an accepted artifact and rate it. If the reviewer does not report a blocker against that criterion, the prompt or the criteria are too weak.
  • Compare against a human rating. For a sample of handoffs, have a person rate the same version with the same rubric before reading the agent’s rating. Where the two disagree by more than a point on a dimension, tighten that dimension’s criteria. Evals for coding agents cover turning these samples into a regression suite.

What breaks in a rating loop?

Section titled “What breaks in a rating loop?”

The score rises but the artifact does not. The reviewer grades the fixer’s notes instead of the artifact. Recover by rating from the frozen version only, diffing the version identifiers between rounds, and treating any fixed finding without evidence in the current version as a blocker, as the re-rate prompt does.

Everything receives 9 or 10. The criteria are too loose to fail. Recover by running the seeded-defect check, adding negative states and failure paths to acceptance.md, and lowering the cap for unsupported claims in the prompt.

The reviewer edits the artifact. The review ran with write access, usually in the producing session. Recover by discarding the edits, rerunning the rating with the read-only command, and keeping fix work in its own task.

Findings disappear between rounds. A fresh reviewer does not know what the last one found. Recover by always passing the ledger into the re-rate, and by never deleting a ledger row.

The loop never converges. Each round fixes one finding and creates another. Recover by stopping at the round cap, reading the last two rating files side by side, and asking whether the criteria contradict each other or the artifact needs a decision from its owner.

Taste outweighs function. Rounds go on spacing and wording while a state is missing. Recover by fixing blockers on intent, states, and accessibility first, and deferring minor system findings to a later round.

Check the rating before the handoff

Section titled “Check the rating before the handoff”
  • The rating file names the exact artifact version it scored.
  • Every fail and not-tested criterion cites evidence another reviewer can reproduce.
  • The jq gate passes on the last rating file.
  • Every finding in the ledger has a status, and every accepted row names its owner.
  • The last rating ran in a fresh, read-only context.
  • The accepted rating and ledger are attached to the next stage’s artifact.

Where to go next from the rating loop

Section titled “Where to go next from the rating loop”
All 577 guides Free resources: Compare tools Glossary By role

Written by

See what's new on the platform.

Privacy Policy Terms of Service Cookie settings llms.txt pricing.md RSS