AI code review test: Greptile vs CodeRabbit
Greptile flagged all 8 planted bugs in each of 3 runs (24 of 24). CodeRabbit flagged 7, 6 and 7 (20 of 24). Neither raised a false alarm.
SR-PR-2 · Nov 8, 2026 · 3 runs per tool · One code-review task (a 63-line pull request with 8 planted bugs), not an overall verdict on either tool.
Result
| Tool | Run 1 | Run 2 | Run 3 | Total | False alarms | Plan |
|---|---|---|---|---|---|---|
| Greptile | 8 of 8 2.7 min | 8 of 8 3 min | 8 of 8 2.8 min | 24 of 24 | 0 | Free plan, standard review |
| CodeRabbit | 7 of 8 9.8 min | 6 of 8 6.8 min | 7 of 8 10.7 min | 20 of 24 | 0 | Free open-source plan (public repo) |
What was missed
- CodeRabbit, run on PR #6: Admin purge route has no admin check.
- CodeRabbit, run on PR #7: Admin purge route has no admin check; Export counter race condition.
- CodeRabbit, run on PR #8: Admin purge route has no admin check.
CodeRabbit's Oct 2026 review of the same pull request scores 8 of 8 under this answer key. Our original round-1 scorer counted 7 of 8; it missed CodeRabbit's race-condition comment because of a keyword match.
How we tested
- Same pull request for every run: 6 files, +63/-5, 8 planted bugs. The answer key was written from the diff before any review was read.
- Each run is a fresh, identical pull request. Default settings, no custom rules or context. Each tool was triggered only on its own pull requests.
- A bug counts when a comment points at the right code and names the actual problem. Real issues outside the 8 are not counted against a tool.
- Times run from the trigger comment to the posted review.
Test pull request: the original change. Answer key: download (JSON).
Caveats
- One small JavaScript pull request. Results on your codebase can differ.
- CodeRabbit ran on its free open-source plan after its trial ended; Greptile ran on its free plan. Both used standard review depth.
- CodeRabbit's free plan allows about one review per hour, so its runs were spread over two hours. Greptile's three ran at the same time.
- Scored by StackRank. Every review comment is public on the linked pull requests, so you can check our scoring.