AI Code Review Bots Compared: Claude Code Review, Codex, Bugbot, CodeRabbit, Greptile and More
PLAI code review bots are second models that read a diff and post findings before a human approves it. The main options in September 2026 are Claude Code Review, claude-code-action, codex review, GitHub Copilot code review, Cursor’s Bugbot, CodeRabbit, Greptile and Alibaba’s open-source ocr. No single bot is enough: run them in layers and measure each on precision and cost per pull request.
Your team merges more pull requests every month, most of them written by agents, and review has become the queue everyone waits in. Someone installs a review bot, it posts 30 comments on the first pull request, and within a week people resolve its threads without reading them. This page is for developers who want a second opinion they can trust before pushing, and for tech leads who have to choose, configure and pay for the bots the whole team lives with.
What you’ll get from this review bot comparison
Section titled “What you’ll get from this review bot comparison”- A comparison table of eleven review options: where each runs, what triggers it, which file tunes it, and what a review costs.
- A four-layer review workflow (local self-review, second model, pull request bot, human) with the order and the cost of each layer.
- The
claude-code-action@v1review workflow adapted from Anthropic’s docs, aREVIEW.mdthat cuts noise, and a workflow that turns the neutral check run into a required status check. - The equivalent wiring for Codex and Cursor, and install commands for Greptile, CodeRabbit and
ocr. - Six copy-paste prompts, four metrics that prove a bot earns its cost, and the traps that make each tool misfire.
- An eval set built from your incident and revert pull requests, a noise measure, a verification pattern and a one-repository rollout.
This page compares the bots. The procedure a human follows on an agent’s pull request is reviewing an agent’s pull request without reading every line; read that first if you have not set up a review checklist yet.
Which AI code review bot fits which job?
Section titled “Which AI code review bot fits which job?”Every tool below was checked on 2026-09-26 against the vendor’s docs, its GitHub repository or npm, or the installed CLIs (Claude Code 2.1.283, Codex CLI 0.157.1). Rows marked secondary rest on search extracts or on a check dated 2026-08-28, not on a vendor page read that day; confirm them on the vendor’s site before you roll them out. The Claude Code Review row and the Claude Code tab were re-read against the Claude Code docs fetched on 2026-10-16.
| Tool | Runs where | Trigger | Tuned by | Cost per review | Evidence |
|---|---|---|---|---|---|
| Claude Code Review (Team and Enterprise) | Anthropic’s infrastructure, GitHub | PR open, every push, or @claude review | REVIEW.md, CLAUDE.md | Billed as usage credits; the launch announcement (9 March 2026) said reviews “generally average $15–25”, and your own average is in Admin settings | Vendor docs and launch announcement |
/code-review (any plan) and claude ultrareview | Your terminal; ultrareview in Anthropic’s cloud | You run it | CLAUDE.md (not REVIEW.md) | Normal usage; ultrareview: 3 free runs on Pro and Max, then “typically $5 to $25” | Vendor docs, CLI |
claude-code-action@v1 | Your GitHub Actions runner | Workflow events you choose | The prompt, the plugin, CLAUDE.md | Tokens on your API key or subscription | Vendor docs |
codex review and openai/codex-action@v1 | Your terminal or runner | You run it; workflow events | Custom instructions, AGENTS.md | Codex usage limits or API tokens | CLI help, action README |
@codex review on GitHub | OpenAI’s cloud | PR comment | AGENTS.md | Codex plan | Secondary (checked 2026-08-28) |
| GitHub Copilot code review (paid Copilot plans; not Copilot Free) | GitHub; its agentic context gathering runs on Actions runners | Request Copilot under Reviewers, or an automatic-review setting or ruleset | .github/copilot-instructions.md, .github/instructions/*.instructions.md, AGENTS.md | AI credits: an estimated $0.05–1 per review at Lite effort, $0.25–5 at Balanced, plus Actions minutes | github/docs |
| Cursor Bugbot | Cursor’s cloud, GitHub and GitLab | PR open or update, bugbot run | .cursor/BUGBOT.md | Usage-based; Cursor says the average run costs $1.00 to $1.50 (checked 2026-10-12) | Cost: Cursor docs and blog; trigger and BUGBOT.md: secondary |
| CodeRabbit (app and CLI) | Vendor cloud; CLI locally | PR; coderabbit review | Vendor config | Not verified | CLI flags verified from CodeRabbit’s own skill |
| Greptile (app and CLI) | Vendor cloud; CLI locally | PR; greptile review | --instructions | Not verified | npm greptile 3.6.0 README |
Alibaba Open Code Review (ocr) | Your terminal or runner, your model | You run it | Rule templates | Your model’s tokens | GitHub README, npm 1.12.9 |
| Graphite Agent (formerly Diamond, Cursor-owned) | Graphite’s cloud | PR | Vendor config | Not verified | Secondary |
Three differences matter more than the feature lists:
- Who pays and how. The managed bots bill per review or per seat; the CLI and Actions routes bill tokens against a key you already own. Token prices live on the models hub, not here.
- Whether it can block a merge. A Claude Code Review review “never approves or blocks the pull request”, and its check run never concludes as a failure (Claude Code docs, read 2026-10-16). Copilot leaves a “Comment” review by default, which does not count toward required approvals; only with Copilot approvals (public preview) switched on can it approve. A gate is something you build from the bot’s output: require conversation resolution on the branch, or read the severity count in CI as shown below.
- What the bot reads for rules. Each bot has its own file. A rule you put in
.cursor/rulesnever reaches Bugbot, and a rule inREVIEW.mdnever reaches the local/code-review.
How does layered AI code review work?
Section titled “How does layered AI code review work?”One bot run by the same model that wrote the code shares that model’s blind spots. Layers work because each one catches a different class of problem at a different price, and each one is cheaper than the next. The tool-specific setups are in code review with Codex and automated code reviews in Claude Code; the team-level review contract, with focused passes for correctness, security, tests and spec compliance, is in govern layered pull-request review.
-
Local self-review, before you push. Run the author tool’s own review in a fresh context:
/code-reviewin Claude Code,codex review --uncommittedin Codex, or the Cursor agent with the first prompt below. It takes seconds to minutes within your normal usage and removes the obvious bugs before anyone else sees them. -
A second model, still local. Ask a different vendor’s model for an adversarial pass:
/codex:reviewor/codex:adversarial-reviewfrom Claude Code,greptile review,coderabbit review --agent, orocr review. A different model family disagrees with the author for reasons of its own, which is the point. -
The pull request bot, once per pull request. One bot owns the pull request: Claude Code Review, a
claude-code-actionworkflow,@codex review, Copilot code review, or Bugbot. It posts inline comments with severities, and CI reads its output to decide whether the merge is blocked. Review on every push is an opt-in for high-risk pull requests, because it multiplies the cost. -
A human, on escalation classes only. A named person reads the code for authentication, payments, migrations and anything else on your escalation list, and signs off on everything else from the evidence. The escalation list and the verdicts are in agent PR review.
What a layered review costs per pull request. Layers 1 and 2 run on usage you already pay for. Layer 3 is the line item: with Claude Code Review at the $15–25 average its launch announcement (9 March 2026) gave, a team merging 200 pull requests a month with one review each spends roughly $3,000–5,000 a month, and “after every push” mode multiplies that by the pushes per pull request. That is arithmetic on Anthropic’s average, not a measurement, so read the per-repository average cost in the admin settings after the first month. A claude-code-action or codex-action workflow bills only tokens plus runner minutes, and you set the model and effort, so it is the cheaper layer 3 when you can live without the managed verification step.
Set up a pull request review bot in each tool
Section titled “Set up a pull request review bot in each tool”The workflow above is the same in every tool. The wiring differs, so pick your tab.
Managed Code Review (Team and Enterprise). An organization Owner enables it at claude.ai/admin-settings/claude-code, installs the Claude GitHub App, selects repositories, and chooses a trigger: once after PR creation, after every push, or Manual. To verify setup, open a test pull request; a check run named Claude Code Review appears within a few minutes. On a Manual repository, comment @claude review for one review, or @claude review always to subscribe the pull request to later pushes. It is not available to organizations with Zero Data Retention or with the HIPAA configuration.
Your own workflow, on any plan with an API key or OAuth token. This workflow is adapted from Anthropic’s GitHub Actions docs (we add persist-credentials: false). Save it as .github/workflows/claude-code-review.yml:
name: Code Reviewon: pull_request: types: [opened, synchronize, ready_for_review, reopened]jobs: review: runs-on: ubuntu-latest permissions: contents: read pull-requests: read issues: read id-token: write steps: - uses: actions/checkout@v7 with: fetch-depth: 1 persist-credentials: false - uses: anthropics/claude-code-action@v1 with: anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} plugin_marketplaces: "https://github.com/anthropics/claude-code.git" plugins: "code-review@claude-code-plugins" prompt: "/code-review:code-review --comment ${{ github.repository }}/pull/${{ github.event.pull_request.number }}" claude_args: '--allowedTools "mcp__github_inline_comment__create_inline_comment"'Two lines decide where the review goes. --comment posts inline comments on the pull request; without it, findings stay in the run log. Keep the claude_args line: the action starts the inline-comment MCP server only when --allowedTools names it. The workflow skips drafts, closed pull requests and pull requests Claude already commented on. The job triggers on pull_request, not pull_request_target, so on public repositories GitHub withholds ANTHROPIC_API_KEY from fork pull requests and forks get no review. persist-credentials: false keeps the GITHUB_TOKEN out of .git/config while the agent runs on the pull request’s code. To authenticate with a subscription instead, run claude setup-token locally and pass the token as claude_code_oauth_token.
Local layers. /code-review reviews your branch and uncommitted changes as a background subagent; /code-review high widens coverage, low reports only the most confident findings, --fix applies findings, and --comment posts them to a pull request. For a deep pass before merge, claude ultrareview 482 reviews pull request 482 in the cloud, where “every reported finding is independently reproduced and verified”; --json prints the raw payload.
Local review (Codex CLI 0.157.1). codex review takes either one target flag or custom instructions, never both. In 0.157.1, passing --uncommitted, --base or --commit together with a prompt fails at parse time with the argument '--base <BRANCH>' cannot be used with '[PROMPT]':
# Terminal, from the repository root (Codex CLI 0.157.1)codex review --uncommittedcodex review --base maincodex review --commit 3f9c2ab# No target flag: the instructions alone drive the review, so name the scope in themcodex review "Review the changes on this branch against main. Focus on auth and SQL injection."A - in place of the prompt reads the instructions from standard input, not a diff, and the same no-target rule applies. For CI, codex exec review takes the same targets with the same restriction, plus --json, --output-schema FILE and -o FILE, which writes the final message to a file a later step can post or parse:
# CI step, after checking out the pull request with full history (Codex CLI 0.157.1)codex exec review --base origin/main -o review.mdYour own workflow. openai/codex-action@v1 installs the CLI and runs codex exec with your prompt; its README ships a complete pull request review example that posts the final message as a comment. For a review job, set permission-profile: ":read-only"; the action refuses to start if you also set sandbox, because “the profile and legacy sandbox systems do not compose”. Since v1.13 (2026-10-05), Linux drop-sudo requires setfacl (the acl package) and ACL support on /run (openai/codex-action CHANGELOG, checked 2026-10-12); see headless agents in CI.
On GitHub (secondary). OpenAI’s docs, checked 2026-08-28, describe enabling Codex code review for a repository and then commenting @codex review (or @codex security review) on a pull request, with custom rules in AGENTS.md. Re-check the setup on OpenAI’s site before rolling it out.
Bugbot (secondary). Cursor describes Bugbot as reviewing “pull requests” and identifying “bugs, security issues, and code quality problems” (cursor.com, checked 2026-10-12). According to search extracts of Cursor’s help pages, you connect GitHub or GitLab under the Cursor dashboard’s integrations, enable Bugbot per repository, and it reviews on every pull request open and update. Cursor’s docs now say Bugbot is configured in Automations (checked 2026-10-12), so those steps may not match the current setup; follow Cursor’s Bugbot page. Comment bugbot run to re-run it. Project rules for Bugbot go in .cursor/BUGBOT.md; .cursor/rules/*.mdc files do not apply to it (Cursor’s docs say rules do not apply to Bugbot PR reviews, checked 2026-10-12). Cursor moved Bugbot from a $40-per-seat subscription to usage-based billing for Teams and Individual plans (Cursor blog, 2026-05-11); effort levels (Low, Default, High, Smart) exist only on usage-based Bugbot plans, @cursor remember [fact] on a PR teaches it a rule, and Autofix spawns a Cloud Agent to fix what a review found (Cursor docs, checked 2026-10-12). On Teams and Enterprise, Cursor’s changelog of 2026-09-23 also lists Security Review, which posts one comment per pull request reporting exploitable bugs, and Rollouts, which monitors each pull request from review to production; this page does not compare them, and code review in Cursor covers Cursor’s own bots.
Second model from Cursor. In Cursor, the quickest second model is a new agent chat on a model from a different vendor than the one that wrote the change, given the adversarial prompt below. For a reviewer outside Cursor, use a CLI: greptile skills install writes Greptile’s skills to .agents/skills/, which Cursor reads, and greptile review runs in Cursor’s terminal. npx skills add coderabbitai/skills installs CodeRabbit’s review skill the same way.
Graphite Agent (secondary). Graphite’s AI reviewer, formerly called Diamond, is reported as owned by Cursor since December 2025. Its stacked-PR CLI is npm @withgraphite/graphite-cli (1.8.6); there is no @graphite/cli.
Tune Claude Code Review with a REVIEW.md
Section titled “Tune Claude Code Review with a REVIEW.md”REVIEW.md sits at the repository root and reaches every agent that finds and verifies findings, so a rule there lands more reliably than the same rule in a long CLAUDE.md. Anthropic lists seven patterns; the four that cut the most noise are redefining Important, capping nits, skip rules and repository-specific checks, plus a re-review rule so a pull request converges instead of collecting new nits on every push. This version, adapted from Anthropic’s example, is a good first commit:
# Review instructions
## What Important means hereReserve Important for findings that would break behavior, leak data,or block a rollback: incorrect logic, unscoped database queries, PII inlogs or error messages, and migrations that aren't backward compatible.Style, naming and refactoring suggestions are Nit at most.
## Cap the nitsReport at most five Nits per review. If you found more, say "plus Nsimilar items" in the summary. After the first review of a PR, postImportant findings only.
## Do not report- Anything CI already enforces: lint, formatting, type errors- Generated files under `src/gen/` and any `*.lock` file
## Always check- New API routes have an integration test- Log lines don't include email addresses, user IDs or request bodies- Database queries are scoped to the caller's tenantKeep it short. Anthropic’s own guidance is that “a long REVIEW.md dilutes the rules that matter most”. The service reads REVIEW.md from the base or default branch and never from the pull request’s own changes, so a change to the file applies only after it merges (Claude Code docs, read 2026-10-16).
Turn the neutral check run into a merge gate
Section titled “Turn the neutral check run into a merge gate”The last line of the Claude Code Review check run’s details is a machine-readable severity count. Anthropic’s launch announcement (9 March 2026) put the average Code Review at around 20 minutes, so the gate has to wait for it rather than poll once. This workflow runs when the check run completes and posts a commit status on the reviewed commit: success when the Important count is zero, failure otherwise. Save it as .github/workflows/claude-review-gate.yml on the default branch:
name: Claude review gateon: check_run: types: [completed]permissions: checks: read statuses: writejobs: gate: if: github.event.check_run.name == 'Claude Code Review' runs-on: ubuntu-latest steps: - name: Turn the severity count into a commit status env: GH_TOKEN: ${{ github.token }} REPO: ${{ github.repository }} RUN_ID: ${{ github.event.check_run.id }} SHA: ${{ github.event.check_run.head_sha }} run: | # The severity line is JSON like {"normal": 2, "nit": 1, "pre_existing": 0}; "normal" = Important important=$(gh api "repos/$REPO/check-runs/$RUN_ID" \ --jq '.output.text | split("bughunter-severity: ")[1] | split(" -->")[0] | fromjson | .normal') || important="" if [ "$important" = "0" ]; then state=success; desc="No Important findings" else state=failure; desc="Important findings: ${important:-unknown}"; fi gh api "repos/$REPO/statuses/$SHA" -f state="$state" \ -f context="claude-review-gate" -f description="$desc"To gate on threads instead, require conversation resolution on the branch: resolving a thread clears a finding and a reply alone does not (Claude Code docs, read 2026-10-16). To gate on the count, mark claude-review-gate as a required status check in branch protection or a ruleset; a check that fails but is not required blocks nothing. The workflow posts a commit status instead of relying on its own job result because, per GitHub’s docs, a check_run workflow runs against the default branch, so its job never appears on the pull request. Until the status arrives, the required check shows as expected and the merge stays blocked, which also covers a review that is still running. On a Manual repository, nothing is posted until someone comments @claude review.
A commit status belongs to one commit and does not carry forward. If the author pushes after the review, the gate on the new tip stays pending and the merge stays blocked until that commit is reviewed. That is the intended behaviour, because an unreviewed commit should not merge, but plan for it: comment @claude review on the final commit before merging, or use @claude review always (or the repository’s review-on-every-push trigger) on pull requests that change after review. Budget for the extra runs, since each review is billed.
For Greptile, greptile review status exits 0 when the commit has a completed review, 3 while one is running, 4/5 for a failed or cancelled review, and 1 when there is none or you are signed out, so a pre-push hook or CI step can require a finished review before anything merges. Treat every code other than 0 as “not reviewed”.
Add a second model: Greptile, CodeRabbit, ocr and the Codex plugin
Section titled “Add a second model: Greptile, CodeRabbit, ocr and the Codex plugin”Each of these gives you a reviewer from outside your author tool. Install commands are verified against each project’s own README or npm entry on 2026-09-26, except CodeRabbit’s installer page, which is secondary.
Codex plugin for Claude Code (OpenAI, openai/codex-plugin-cc). It runs your local Codex install from inside a Claude Code session and counts against your Codex usage limits. /codex:review is “a normal read-only Codex review” and takes no focus text; /codex:adversarial-review is steerable and challenges the design.
# Inside Claude Code/plugin marketplace add openai/codex-plugin-cc/plugin install codex@openai-codex/reload-plugins/codex:setup/codex:adversarial-review --base main challenge whether this retry design is safe under concurrent writes/codex:setup --enable-review-gate adds a Stop hook that runs a Codex review on every Claude response and blocks the stop when it finds issues. The README warns that the gate “can create a long-running Claude/Codex loop and may drain usage limits quickly”, so enable it only in a session you are watching.
Greptile CLI (npm greptile 3.6.0, Node 22 or later). greptile review compares the merge base with HEAD; --plus and --apex raise the effort.
npm install -g greptile # or: brew install greptileai/tap/greptilegreptile # signs you in on first rungreptile init # admin only: enables the repositorygreptile review -b main --instructions "focus on retry cancellation"CodeRabbit CLI. Install it from CodeRabbit’s official installer or Homebrew, not from npm. CodeRabbit’s own coderabbitai/skills SKILL.md points to https://www.coderabbit.ai/cli. The install lines below are secondary: they come from search extracts of CodeRabbit’s docs, and the installer page was not re-read on 2026-09-26. Do not install the npm name: the npm package coderabbit is a “security holding package” (version 0.0.1-security.1). --agent returns findings an agent can act on, with severities from critical to info.
curl -fsSL https://cli.coderabbit.ai/install.sh | sh # or: brew install --cask coderabbit (secondary)coderabbit auth logincoderabbit review --agent --uncommittedcoderabbit review --agent --base mainAlibaba Open Code Review (ocr), Apache-2.0, npm @alibaba-group/open-code-review 1.12.9. It pairs deterministic file selection and rule matching with an LLM agent, and works with any OpenAI- or Anthropic-compatible endpoint, so the code never leaves the provider you already use.
npm install -g @alibaba-group/open-code-reviewocr config provider && ocr config modelocr review --from main --to feature-branch --format json --output result.jsonAlibaba’s README reports higher precision and F1 than Claude Code with the same model, at about one ninth of the tokens, and says recall is lower, “a deliberate trade-off favoring precision over noise”. That is a vendor benchmark (200 pull requests from 50 repositories), not reproduced here. Choose ocr for a CI layer where noise costs you more than a missed nit.
As of 2026-09-26, anthropics/claude-code-action had 8,951 GitHub stars, openai/codex-action 1,248, openai/codex-plugin-cc 33,594 and alibaba/open-code-review 41,400; in Anthropic’s plugin directory, greptile showed 56,611 installs and coderabbit 32,361 (GitHub and claude.com/plugins, read for this site’s ecosystem catalogue). Stars measure attention, not review quality. Both plugins are in Anthropic’s official marketplace (claude-plugins-official, checked 2026-09-26):
# Inside Claude Code/plugin install greptile@claude-plugins-official/plugin install coderabbit@claude-plugins-officialThe plugins carry context cost in every session. Before you keep one, run claude plugin details greptile or claude plugin details coderabbit in a terminal, which shows the plugin’s components and projected token cost, and compare /context before and after.
Copy-paste prompts for layered AI code review
Section titled “Copy-paste prompts for layered AI code review”The third prompt produces the numbers you need for the precision metric below. In Claude Code with the GitHub CLI installed, the agent reads comments with gh api; in Codex and Cursor, use the GitHub MCP server or gh in the terminal.
How do you prove a review bot is worth its cost?
Section titled “How do you prove a review bot is worth its cost?”A review bot is itself an unverified agent until you measure it. Track four numbers per bot per month, and review them with the team:
| Metric | Definition | Act when |
|---|---|---|
| Precision | Findings that led to a code change, divided by findings raised | It falls below the level where people still read the comments; tighten REVIEW.md or BUGBOT.md, or lower the effort |
| Escapes | Production defects in code the bot reviewed, where a comment from the bot would have been enough | It rises; add the missed class to the bot’s “Always check” list |
| Cost per merged PR | Bot spend for the month divided by merged pull requests | It rises faster than merged PRs; move the bot from “every push” to “once per PR” or Manual |
| Time to first review | Minutes from the PR opening to the bot’s first comment | It exceeds your review SLA; Anthropic’s launch announcement (9 March 2026) put Claude Code Review at about 20 minutes on average, so do not make humans wait for it |
The bots never own the decision. CI owns the gate (tests, types, lint, the severity-count check above), the bot owns the comments, and the approver named on the pull request owns the merge. The code owners in CODEOWNERS sign off on escalation classes, whatever the bots said. Canonical definitions for team metrics live in metrics frameworks for agentic engineering.
How do you know a reviewer got better, not just different?
Section titled “How do you know a reviewer got better, not just different?”The four metrics above are a monthly dashboard, and next month’s pull requests differ from this month’s. To know whether a change to the reviewer’s prompts, tools, model or REVIEW.md helped, rerun a fixed set of cases. The Claude Code team’s set (“How the Claude Code team does code review”, Addy Osmani, Morgan Westlee Lunt and Qing Wang, 15 October 2026) is past pull requests that caused incidents, with their fix-forwards and reverts, each bug confirmed by engineers who know the code. The reviewer runs against it daily, every change to the reviewer is scored against all of it, and it grows with every incident. The article does not say how a case is replayed, so the steps below are one way.
-
Collect the cases from your incident tracker and your reverts. The revert titles carry the caveat from the review queue page:
Terminal window # Terminal, inside the repository, gh authenticatedgh pr list --state merged --limit 200 \--json number,title \--jq '.[] | select(.title | startswith("Revert")) | [.number, .title] | @tsv' -
Have someone who knows the code confirm each case: the commit or range that introduced the bug, the file and lines, what goes wrong, and who confirmed it. An unconfirmed case tests the reviewer against your guess.
-
Save the cases next to the reviewer’s configuration (tab-separated, the row is an illustration):
evals/reviewer/cases.tsv case review_range bug_location bug confirmed_byinc-0412 9c1e7a2^..9c1e7a2 src/refund.ts:88 Retry lacks idempotency key @anna -
Replay each case from a checkout of the code as it was. A reviewer that reads files around the diff would otherwise see today’s code, with the bug already fixed, and recall would be inflated. Use a throwaway worktree at the range’s end commit, run the reviewer from it, and save each output in the main checkout as
evals/reviewer/runs/RUN_ID/CASE.md(RUN_IDnames the run,CASEthe case), so that the worktree stays clean andgit worktree removecan delete it:Terminal window # Terminal, main checkout: detached worktree at the end commitgit worktree add --detach ../eval-inc-0412 9c1e7a2# run the reviewer from there; write its output to the main checkoutgit worktree remove ../eval-inc-0412Run the reviewer from that worktree:
/code-reviewin Claude Code: start the session in the worktree and run/code-review 9c1e7a2^..9c1e7a2. It readsCLAUDE.mdand notREVIEW.md, so it does not stand in for the managed service.codex review: runcodex review --commit 9c1e7a2(Codex CLI 0.157.1).- A hosted bot such as Claude Code Review or Bugbot: push the case’s parent and end commits as two scratch branches and open a pull request from the second into the first, so the bot sees the code as of that commit. Then request a review with
@claude revieworbugbot run(secondary).
A hosted replay is billed like any review. Anthropic runs its set daily; run yours weekly until you know the cost.
-
Score recall and count the rest. A case is found when the reviewer posts a finding at the bug’s location that describes it. Recall is cases found divided by all cases; every other finding goes into the noise count below. The first prompt below scores a run.
-
Rescore after every change to a prompt, tool, model, level or rules file. The service never reads
REVIEW.mdfrom the pull request’s own changes (Claude Code docs, read 2026-10-16), so a change to it applies only after it merges to the base or default branch; rescore then. A change that raises recall and noise together is not yet an improvement.
Replace RUN_ID with this run’s folder name and PREVIOUS_RUN_ID with the last run’s.
How do you measure a reviewer’s noise?
Section titled “How do you measure a reviewer’s noise?”A reviewer can raise recall by flagging more of everything. The article: “It’s easy to catch more bugs if you don’t mind flagging a lot of non-bugs, so we need a check on that.” Anthropic sorts every comment by how the person on the pull request responded (fixed it, said it was wrong, said the behavior was intentional, or decided it didn’t matter), and runs the reviewer on pull requests known to be clean, counting reports that would have led someone to make a harmful change.
The article does not say how responses are recorded. One cheap convention: start each reply with fixed:, wrong:, intentional: or minor:, which the prompt below tallies. Each response points at a lever:
| What the author did | What it tells you | What to change |
|---|---|---|
| Fixed it | A useful finding; their share is the precision metric above | Keep the rule that produced it |
| Said it was wrong | A false positive | Require a file:line citation for that class of finding, or add a skip rule |
| Said it was intentional | The reviewer lacked context | Put the context in CLAUDE.md or the rules file |
| Decided it didn’t matter | True, and not worth a round | Cap the nits and raise the bar for Important |
For the clean set, use merged pull requests that nobody later reverted, fixed forward or tied to an incident, and replay them the same way, from a checkout at each pull request’s head. The recall score hides the harmful reports you count there.
Anthropic’s written priority is real bugs first, fewer review rounds second. Write yours the same way: a change that cuts rounds by missing known bugs then fails the eval, and one that finds more at the cost of more rounds is a trade you make on purpose.
How often a reviewer speaks is separate from how often it is right. At the research-preview launch, the share of Anthropic’s internal pull requests with substantive review comments went from 16% to 54% (launch announcement, 9 March 2026, repeated in the 2026-10-15 article); most pull requests over 1,000 lines get findings and most small ones come back clean. That is a coverage figure about its own pull requests, so read it next to the noise measures.
Noise costs more when the author is an agent: a person waves off a nit in seconds, while an agent fixes it, which means another commit and another round. Nit caps and “Important only after the first review” rules in REVIEW.md matter most there; the round cap covers the rest.
How does verification by voting cut a reviewer’s noise?
Section titled “How does verification by voting cut a reviewer’s noise?”In the article, several agents look for bugs at once, then a separate group votes on each finding, and a finding is posted only if enough of them agree it is real. Style-, diagnostics- and cleanup-only findings are dropped, filters the team A/B tested on its own pull requests. What remains is tagged Important, Nit or Pre-existing, “so a bug the PR didn’t introduce doesn’t look like the author’s fault”. The Claude Code docs describe the public service as find, verify, rank and post, with verify filtering out false positives (read 2026-10-16); the voting detail is the article’s. For your own reviewer as a subagent or skill, copy the shape:
- Separate the finder from the verifier. The verifier sees one finding and the code it points at, without the finder’s reasoning, and tries to refute it.
- Post only what survives: a majority of several verifiers, or one verifier’s
file:lineevidence. Drop style-, diagnostics- and cleanup-only findings unless the style hides a bug. - A/B test each filter on the eval set before you keep it. Recall must not fall, and noise must.
How do you roll out and tune a reviewer?
Section titled “How do you roll out and tune a reviewer?”The article’s advice is one repository first, a few weeks of tuning, then widening:
-
Start on one repository where pull requests are frequent and a missed bug has a cost. For Claude Code Review, the costs page suggests a pilot with a monthly spend limit and Review runs on Once after PR creation or Manual, then reading each repository’s average cost per review (Claude Code docs, read 2026-10-16).
-
Write the rules file short. The article’s good
REVIEW.mdfiles say what failure looks like in that repository (for docs, “it lies”; for config, “it misroutes”), cap the nits, list paths to skip, ask for a file and line as evidence, and ask for a summary that starts with a count and says when there is nothing to fix. The same traits suitAGENTS.mdfor@codex reviewand.cursor/BUGBOT.mdfor Bugbot. The eval above checks that the reviewer follows the file. -
Match the level to the cost of a miss. An Owner sets Claude Code Review to Quick, Standard or Deep, with a per-repository override; “Deep is worth it where one missed bug costs more than the extra review time”. Bugbot has effort levels (Low, Default, High, Smart) on usage-based plans, as the Cursor tab above records.
-
Each week, fix repeats at the source: the skill, a lint rule or the instruction files. When the reviewer misses something, ask what it did not have in front of it and put that into
CLAUDE.md, a skill or a check. Then rescore the eval and read the response table. -
Widen after a few weeks, starting the next repository from the first one’s rules file and eval cases.
What breaks when you run AI code review bots?
Section titled “What breaks when you run AI code review bots?”The bot drowns the real bug in nits. Thirty comments, one of them important, and the team stops reading. Recovery: cap nits in REVIEW.md or BUGBOT.md, add “after the first review, post Important findings only”, and track precision. If precision keeps falling, switch that bot to Manual.
Two bots argue on the same pull request. Claude Code Review and Bugbot both comment, disagree, and the author fixes one and reopens the other. Recovery: one bot owns layer 3. Run the second model locally (layer 2), where its output goes to the author, not to the pull request.
Push-triggered reviews multiply the bill. A pull request with 12 pushes gets 12 reviews. Recovery: use “once after PR creation” or Manual, and ask for @claude review always only on high-risk pull requests. Set a monthly spend cap for Claude Code Review at claude.ai/admin-settings/usage.
Forks and drafts get no review, and nobody notices. GitHub withholds secrets from fork pull requests, claude-code-action skips drafts, and Code Review reviews forks only on a comment command. Recovery: do not rely on the review job as a required check, because GitHub counts a skipped job as passed. Require the claude-review-gate status above instead: it stays pending, and blocks the merge, until the tip commit has a review. Have a maintainer comment @claude review on fork pull requests before approving.
The team treats a clean bot run as approval. “The bot found nothing” becomes the merge reason. Recovery: gate on tests and the severity count, leave Copilot approvals off unless a human still signs off on escalation classes, and keep a named human approver on every pull request.
A change raises recall and noise together. The eval shows more known bugs found, and nobody counted the extras. Recovery: score both on every run, and keep the change only if your written priority allows the trade.
The eval scores the reviewer’s opinion. Cases nobody who knows the code confirmed reward whatever the reviewer already says. Recovery: have a person who knows that code confirm every case, as Anthropic’s set does, and drop any case nobody can confirm.
A review loop drains usage. The Codex plugin’s review gate or an autofix bot keeps finding and fixing new issues. Recovery: enable the gate only in watched sessions, stop at a round cap (three rounds in Anthropic’s setup, two returns on the agent PR review page), and escalate to a human with the open findings listed; the round cap section explains why later rounds mostly find old code.
Where to go next with AI code review
Section titled “Where to go next with AI code review”Did this guide help you?
One tap, no sign-in.
What will you do with it?
How did it read?
What got in the way? Pick any that apply
Your role?
Thanks. I read every answer. Michał Jaskólski
Did this guide help?
One tap each. Close whenever you like.
no sign-in