CodeRabbit Bake-Off Evaluation

Contents24 sections

review:tool-bakeoff phase-3 decision record (compiled 2026-07-27, two days before the ~2-week trial window closes ~07-29). Grades CodeRabbit on the same seven metrics used for Gemini Code Assist and Qodo Merge — findings/types · speed · efficiency · learning · cost · quality — over the full 2026-07-16 → 07-27 window across all four repos in the MAX Research Collective, then runs the direct three-way comparison, states pros/cons, makes the keep-or-dump call, and names who to evaluate next.

Fleet scope: MAX3 (newellnarco/max3) · REEF (newellnarco/netsniff) · FeedHacker (newellnarco/feedhacker) · MAX Research Collective site (newellnarco/maxresearchcollective).

Companions: PR_REVIEW_BAKEOFF.md · MAX_COLLECTIVE_FINAL_OUTCOME.md · AI_REVIEW_FINDINGS_LEDGER.md · CODE_RABBIT_ONBOARDING.md · MODEL_COUNCIL.md.

Provenance (Rule 4/10)

This record is compiled from a four-repo data sweep (2026-07-27): each repo's own durable ledger (AI_REVIEW_FINDINGS_LEDGER.md on MAX3; KNOWN_ISSUES.md on REEF + FeedHacker — the "close the loop → ledger row with a Found by attribution" rule makes these the complete attribution record), plus GitHub-API enumeration of the window's PRs, CodeRabbit engagement, and first-response latency samples. Every headline number below is measured; every gap is flagged.

The three phases:

• Phase 1 (07-05→11): Gemini vs Qodo out-of-box — 394 findings hand-rated across 110 MAX3 PRs.
• Phase 2 (07-13→16): tuned Qodo vs sunsetting Gemini.
• Phase 3 (07-16→27, this record): CodeRabbit live across the fleet under one tuned config

family. Gemini ceased 07-17; Qodo wound down. REEF + FeedHacker are independent cross-repo replications; MRC is the deliberate negative control (a static site with CodeRabbit correctly OFF).

The full-window numbers (measured)

Fleet volume + coverage, 2026-07-16 → 07-27:

• The un-engaged code-repo PRs are almost entirely docs/board fast-path PRs CodeRabbit is

configured to skip (!**/*.md path filters + docs:/[auto]/board: title keywords). Effective coverage of code PRs on the two high-volume repos is materially above the all-PR figure.

• Across the fleet CodeRabbit engaged on 193 PRs in 11 days — this is not a trial dabble, it is

the production reviewer on three repos.

Cross-reviewer head-to-head (REEF's KNOWN_ISSUES.md, the cleanest apples-to-apples — 189 total ledger rows, counted per bug row not line-mentions):

The three bots never ran concurrently as primary — Qodo/Gemini owned the earlier ~#58–115 span, CodeRabbit took over at #102 and is solo from ~#116. So over a comparable ~100-PR tenure, CodeRabbit out-produced Qodo (67 vs 46) and Gemini (67 vs 25) on sole-credit real bugs — while running alone.

Findings & types

MAX3 (the AI_REVIEW_FINDINGS_LEDGER roll-ups — all CodeRabbit):

• Signal ratio improves over the window: declined+FP fell from 3/17 (roll-up 001) to 0 in both

later roll-ups. One hard false positive all window (the sanctioned [auto] loop-guard "correction", fixed at source via a path_instructions exception).

• The ~39 is a floor — the ledger records the roll-up windows + #1314, not a full census of all 87

engaged PRs.

Types it lands (the value shape) — reproduced on three stacks. The recurring, highest-value class across the fleet is false-green / dishonest-on-failure — a status/diagnostic/watch reporting alive/ok/"up to date" when the underlying check errored, was empty, was loading, or was stale:

• MAX3: fail-closed rendering (roll-up 002, 8× one class → drove 2 ast-grep lints); HTTPError-first

exception ordering (the "earns-its-keep" correctness catch); a SKOS export shipped as a custom shape not real JSON-LD (#1392); a verify_schemas if/elif short-circuit under-reporting a both-domain-and-range violation (#1390); a telemetry decide/commit-on-success retryable-state bug (#1314).

• REEF (the security register): a firewall block-guard verifier reading a present-but-unreadable

firewall as green (#190); the /api/fleet/enable JSON-bool trap (bool("false") truthy → a real settings write, #183); a Pi-hole v6 auth bug that would 401→silently-dark + unknown status defaulting to "blocked" (#181); unattended auto-resolve hazards (irreversible ops must go manual, no action on a degraded feed, PIN-gated undo must refuse with no PIN set, #157); a liveness strip's stale-green + a11y gaps (#182). ~21 of REEF's 70 CodeRabbit rows touch the security/PIN/firewall/auth/fail-closed surface.

• FeedHacker: a persisted-chrome.storage migration bug — removed toggles didn't reset, so old

non-default choices survived for existing users (#47, Major); a screen-reader double-announce on decorative SVGs (#47); and — like MAX3's #1302 — its own .coderabbit.yaml schema error on the first review (#45). Its ast-grep essential_rules also fired a real inner-HTML/XSS CWE-79 flag.

Verdict: real bugs in the highest-value register (fail-closed, irreversible-action guards, auth, races, config self-review) — not lint — reproduced on a Python AI resident, a Next.js network console, and a TypeScript browser extension. Signal ratio is high and improving (0 noise on 2 of 3 MAX3 roll-ups; 1 true FP all window; 0 declined on REEF/FeedHacker samples).

Speed

• First-response latency ≈ 2–6 min on the two high-volume repos (MAX3 sampled median ~2.7 min:

#1392 ~2.0m · #1379 ~1.7m · #1314 ~3.4m · #1390 ~4.9m; REEF: #182 ~3.1m · #181 ~6.1m). Well inside the 10-minute CI hard gate.

• FeedHacker is slower and variable (~7 min to ~1h46m) — a small repo with asset-heavy diffs; latency

scales with diff size, not a defect.

• Reviews drafts at open (drafts: true) — the load-bearing setting; all three code repos hit

CodeRabbit's default draft-blind-spot and fixed it identically. Adds pre_merge_checks (a gating read Gemini/Qodo didn't surface).

• Incremental re-review fires within minutes of each push (auto_incremental_review: true +

auto_pause_after_reviewed_commits: 0) — confirmed working on REEF #187 (3 rounds).

Efficiency — signal ÷ effort

• 0 Actions minutes (hosted app) on all repos — the reviewer choice adds nothing to the separate

~$250/mo CI Actions overage.

• chill profile + diff-scoped → low noise (≈0 on 2 of 3 MAX3 roll-ups; only 1 trivial nitpick + 1

true FP measured all window).

• The one measured drag is rate-limiting on MAX3 (roll-up 001 ~6×; roll-up 002 13-min waits, ~15

reviews / 9 PRs; roll-up 003 ×2 on #1392) — author-side, because every incremental commit re-triggers a full review and mid-review pushes cause "head changed" restarts. No rate-limit surfaced on REEF or FeedHacker in the samples. It is a shift-left signal, not a coverage loss (no finding was ever dropped by the limit) — the fix is review:shift-left-local (run the local pass / ModelCouncil before opening the PR).

• Fast-merge hole (REEF): real but narrow. Median REEF PR lifetime 13 min, 55/100 merged ≤15 min —

yet 96% still got a review (first-review ~3–6 min beats most merges). One documented in-window miss:

#161 merged on the walkthrough before its 10 findings landed (fixed as a follow-up). Mitigation is the ≥15-min / review-arrived merge gate.

Learning — the differentiator

CodeRabbit is the only candidate with two compounding learning surfaces, and it is the axis where it beats both predecessors outright:

The flywheel is live and measured:

• REEF best_practices.md grew 717 → 1,945 lines (+1,228, 2.7×) over the window — its most-repeated

CodeRabbit classes (§60 test-quality, §63 debounce, §64/65 honest-diagnostic, §67 fleet-light-law) were all added from CodeRabbit findings in-window.

• MAX3 shipped 3 repo-local ast-grep lints, 2 of them (no-synthesized-green-status,

no-mock-usepanelfetch-in-states-test) generated directly from recurring CodeRabbit findings (roll-up 002) — turning the fleet's #1 class into an authoring-time, $0 catch.

• FeedHacker's 164-line best_practices.md grew §19–31 from CodeRabbit/REEF feedback.

Gemini could not learn at all; Qodo learned from accepted suggestions but had no structural-lint layer.

Diagram B — The learning flywheel: a recurring finding becomes a rule + ast-grep lint, then a $0 authoring-time catch.

Cost

• Plan in use across the three code repos: CodeRabbit "Pro Plus" (trial), profile CHILL (confirmed in

review-footer metadata). CodeRabbit Pro's unlimited is the standard plan (~$24–30/mo, 1 seat), no promo cliff — normalized ~$0.09–0.11 per reviewed code PR at fleet volume.

• One newellnarco seat is per-account, not per-repo — it covers all four repos at once. (The exact

Pro-vs-Pro-Plus monthly price and what Pro Plus adds should be re-verified at signup — see Limits.)

• No promo cliff — Qodo's headline risk (its "unlimited" is a time-limited promo over a documented

20-PR/seat/month standard, which at ~450–600 org PR/mo implies ~23–30 phantom seats and the ~$2,040/mo vendor quote).

• Only recurring cost in-window was attention + MAX3 rate-limit waits from draft re-pushes (§3) — a

process fix, not an invoice.

Quality

On the one class that matters most (dishonest-on-failure), CodeRabbit is a first-class catcher across three stacks, its findings are grounded (cite best_practices.md rules + prior PRs; #1302 cited docs.x.ai), and it reviews and corrects its own config (MAX3 #1302; FeedHacker #45; the two config gotchas caught on REEF #102). Signal ratio: improving to 0 noise; 1 true false positive all window.

Honest cons: the [auto] false positive is real reviewer time-waste; MAX3 rate-limit friction is real; and no clean same-code three-way ran in phase 3 (Gemini dead, Qodo wound down) — so phase-3 quality is judged against the phase-1/2 baselines + the REEF/FeedHacker replications, not a fresh identical-diff head.

Direct comparison — Gemini vs Qodo vs CodeRabbit

Read: Gemini was fast but stale (its last-day #1302 review was 4/4 wrong on CodeRabbit's own config schema and it conceded on-thread) and is dead. Qodo tuned was excellent but carries a pricing cliff and no structural-lint learning. CodeRabbit reproduces the tuned-Qodo grounding, out- produces both on sole-credit real bugs over a comparable tenure, adds the ast-grep learning layer nobody else has, prices at flat unlimited-standard, and is already the production reviewer on three repos under one config.

Fleet view — CodeRabbit across all four repos

One tool, one config family, right-sized per stack — and correctly off on the one repo where a code reviewer adds nothing:

Diagram A — Fleet decision map: one per-account seat drives three code repos; MRC off by design; capped at two lanes.

• The method generalizes across stacks. The #1 class (false-green / dishonest-on-failure) and the

config-self-review moment reproduced on all three code repos — a Python AI resident (MAX3), a Next.js network console (REEF), and a TypeScript browser extension (FeedHacker). Not a MAX3 artifact.

• Lessons are shared, not re-learned. Every code repo carries the same load-bearing config —

drafts: true (all three hit the draft-blind-spot and fixed it identically), profile: chill, auto_pause_after_reviewed_commits: 0, docs/board title-skips, pre_merge_checks docstring-coverage OFF for non-JSDoc code, and best_practices.md grounding. One config investment amortizes across the fleet.

• The learning surfaces scale with the repo. REEF's 1,945-line best_practices.md and MAX3's 3

ast-grep lints are where it earns most; FeedHacker's 164 lines match its smaller volume.

• MRC is the correct negative control. In 32 window PRs CodeRabbit produced 0 reviews, 0 findings,

1 draft-skip status — the repo .coderabbit.yaml disables it (enabled:false, committed 07-20). A no-build static marketing site has no runtime failure paths for a reviewer to reason about. The one honest caveat: a minority of MRC PRs touch real logic (the in-browser main.js retrieval, the Python build_site.py, and the SFTP/SMTP-secret GitHub workflows) — if that surface grows, selectively re-enabling review there is worth revisiting. As of this window, OFF is right.

• Fleet economics. CodeRabbit Pro's seat is per-account — one newellnarco seat (~$24–30/mo)

covers all four repos. This is why the identity rule bites (FeedHacker CLAUDE.md): CodeRabbit is single-user-licensed to newellnarco, so a PR under any other login is silently unreviewed — the whole fleet loop depends on the author being newellnarco (confirmed 100% across every repo's window PRs).

Recommendation — KEEP CodeRabbit; DUMP the rest (fleet-wide)

KEEP CodeRabbit Pro (1 newellnarco seat, ~$24–30/mo) as the primary hosted reviewer on the three code repos — MAX3, REEF, FeedHacker; keep it deliberately OFF on the MRC static site. One per-account seat covers all four. It wins on five of seven axes (findings/output, learning, cost, availability, extras), ties on efficiency/Actions, is only behind on raw first-response latency vs a now-dead tool — and the full-window data shows it out-produced both predecessors on real bugs over a comparable tenure. Its ast-grep learning layer directly serves the CICD_CONTINUOUS_IMPROVEMENT §2b / AI_REVIEW_FINDINGS_LEDGER flywheel, reproduced on three stacks.

• DUMP Gemini — already gone (07-17 sunset); its only lost value (docs coverage) is the lowest-value

slice, owned at $0 by docs:drift-lint.

• DUMP paid Qodo as a second lane — CodeRabbit's unlimited-standard + ast-grep + learnings:local +

built-in gitleaks cover Qodo's grounding value without the promo-cliff, and one config serves all four repos. Keep the tuned .pr_agent.toml / best_practices.md assets as sunset-proof break-glass, not a paid subscription.

• KEEP the on-box ModelCouncil as the free second voice — the whole bake-off proved two near-disjoint

reviewers (~2% overlap) are purely additive; the council restores Gemini's 2-min first-pass at $0 on our own keys, no seat/quota/sunset. Wire it onto PR diffs (the standing "say GO" item).

Net stack: 1 paid hosted lane (CodeRabbit ~$30/mo, one seat covering all four repos) + $0 on-box ModelCouncil.* Cap at two lanes — the severity-4 tail is paid in attention*, the scarce resource in a shop of one.

Conditions on the keep (the shift-left obligations)

The one measured cost in phase 3 was MAX3 rate-limit friction from draft re-pushes. Keeping CodeRabbit means adopting the shift-left discipline already promoted in the ledger: run the local pass (coderabbit review --plain / ModelCouncil) before opening the PR; ship standards-correct + the full unit/integration/system triad in the first commit; batch fix-commits, never re-push mid-review; and adopt the ≥15-min / review-arrived merge gate (the REEF fast-merge hole). These collapse N hosted rounds → 1 and stop both the rate-limit spend and the merge-before-review miss.

Others to evaluate next (quarterly rotation for the second lane)

The second lane is the free on-box ModelCouncil today. Rotate a challenger through a future quarterly bake-off; from the verified landscape (MAX_COLLECTIVE_FINAL_OUTCOME Appendix E), the fitting flat-rate candidates are:

Reject (permanent) at this volume: Copilot code review (~$130–190/mo, ~$0.52/review), Greptile v4 (~$260–330/mo), Cursor Bugbot (~$280–525/mo) — all per-review-priced, 10–20× flat-rate for an autonomous loop. Amazon Q is a same-vendor rug-pull risk (sunset for Kiro).

Limits of this evaluation (Rule 4)

• The ~39 MAX3 actionable count is a documented floor, not a full census — a complete count needs

get_reviews on all 87 engaged PRs. REEF's ledger records one row per PR (a 10-gap PR = 1 row), so its 67-row count also understates raw finding volume.

• Per-PR latency + findings-per-PR are sampled (3–8 PRs per repo), not aggregated fleet-wide.
• No clean same-code three-way ran in phase 3 (Gemini dead, Qodo wound down); CodeRabbit is judged

against the phase-1/2 baselines + the REEF/FeedHacker replications.

• MAX3's KNOWN_FAILURE_PATTERNS.md has no "Found by" tagging, so MAX3 can't be counted the way

REEF/FeedHacker are — its number comes from the findings-ledger roll-ups.

• Plan/price: the review footers show "Pro Plus" (trial); the exact standard monthly price and what

Pro Plus adds over Pro must be re-confirmed at signup — the decision assumes unlimited stays the standard tier (the whole reason it beats Qodo).

Cross-references

• PR_REVIEW_BAKEOFF.md — phases 1–3 running ledger.
• AI_REVIEW_FINDINGS_LEDGER.md — the CodeRabbit findings roll-ups.
• MAX_COLLECTIVE_FINAL_OUTCOME.md — measured output + full pricing/landscape.
• CODE_RABBIT_ONBOARDING.md — the tuned .coderabbit.yaml + ast-grep config.
• MODEL_COUNCIL.md — the free on-box second voice.

Fig 1 — Cross-reviewer real-bug output (…_solecredit.png)

Source: REEF docs/KNOWN_ISSUES.md, sole-credit bug rows (one row per bug), over comparable tenures.

Fig 2 — Fleet coverage (…_coverage.png)

Source: GitHub-API enumeration of PRs created 2026-07-16..07-29 + commenter:coderabbitai[bot].

Fig 3 — best_practices.md growth (…_flywheel_growth.png)

Source: line counts of each repo's best_practices.md (REEF before/after the window).

REEF delta: +1,228 lines (2.7×) in 11 days.

Fig 4 — MAX3 signal ratio (…_signal.png)

Source: AI_REVIEW_FINDINGS_LEDGER.md roll-ups (all CodeRabbit).

Fig 5 — First-response latency (…_latency.png)

Source: GitHub-API timestamps (PR open → first CodeRabbit review), sampled PRs.

Reference lines: Gemini median 2.2 min; CI hard gate 10 min.

Fig 6 — Normalized cost per reviewed code PR (…_cost.png)

Source: MAX_COLLECTIVE_FINAL_OUTCOME.md Appendix F, at measured fleet volume.

Fig 7 — Seven-metric three-way (…_radar.png)

Working 0–5 rubric (one rater, derived from the metric sections above — not a vendor metric; the same "working rating" convention as the phase-1 severity scale). 0 = absent/worst, 5 = best.

Rationale (grounded in the sections): Gemini scores Speed 5 / Cost 5 but Learning 0 / Availability 0 (dead 07-17 + no learning surface); Qodo Cost 2 (promo-cliff) / Learning 3 (reads standards, no structural lint); CodeRabbit Findings 5 (out-produced both) / Learning 5 (standards + threads + ast-grep) / Availability 5 (unlimited-standard, no cliff), Speed 4 (~2–6 min, behind Gemini's 2.2).

Diagram A — Fleet decision map (…_fleet_map.png)

Layout data (no numeric series): one per-account CodeRabbit Pro seat → 3 code repos PRIMARY (MAX3, REEF, FeedHacker) + MRC OFF (dashed); the on-box ModelCouncil as the $0 second voice; DUMPED = Gemini (dead 07-17) + Qodo (pricing-cliff). Cap: two lanes.

Diagram B — The learning flywheel (…_flywheel.png)

Cycle: (1) CodeRabbit finds a recurring class → (2) promote the prevention to a best_practices.md rule + an ast-grep lint → (3) caught at authoring, shift-left, $0 — the hosted round is not spent → (4) fewer hosted findings of that class next PR, so the reviewer's spend shifts to higher-value catches → back to (1).