Lessons from a Week Evaluating an AI PR Reviewer
Lessons from evaluating an AI PR review plugin: fixing the wrong skill, why risk classification matters, and how upstream evidence quality shapes review accuracy.
This piece documents the evaluation of an AI-powered PR review plugin, revealing that an unguided Claude Opus baseline already catches roughly 65-70% of textbook bugs on its own. The plugin's real value turns out to be suppressing false positives and improving risk classification rather than finding more bugs, since the base model already handles most easy wins.
One of the most striking findings came from changing evaluation scoring weights without touching any code: the gap between the plugin and baseline widened by 9 percentage points simply because risk classification, something the plugin was already doing correctly, had been underweighted in the original scoring.
The most instructive episode involves a PR that added authorization guards to previously unguarded routes, which the reviewer kept flagging as high-risk despite four rounds of prompt rewrites. The real issue was upstream: the evidence builder had already classified the change as 'red lane' before the reviewer ever saw the diff. A single paragraph fix in the evidence builder's classification logic solved it instantly, with zero changes to the reviewer itself.
The broader lesson for engineers building multi-stage AI systems is that debugging the wrong component can waste significant time. Downstream quality is bounded by upstream evidence quality, making it more valuable to locate the correct layer to fix than to endlessly tune prompts.