The 75% False-Negative Wall LLM Reviewers Can't Move
Why LLM-based reviewers hit a fixed 75% false-negative rate, and why voting, reruns, and prompt calibration all fail to move it.
A series of experiments testing LLM-based review of AI agent outputs uncovered a structural limit: once a reviewer gets sharp enough to catch all obvious garbage (empty test runs, TODO placeholders, nonsense text), it also rejects three out of four legitimate outputs. This 75% false-negative rate held steady across different models — GLM-5.2 and deepseek-v4-flash both hit it — and across test sets scaled from 8 to 30 scenarios.
Three standard fixes were tried and none moved the wall. Majority-voting reruns of the same prompt didn't help because the model gives the same wrong verdict every time. Voting across strict/balanced/lenient prompt variants didn't help because all three share the same underlying bias. Prompt calibration appeared to work on a small 8-scenario set, but at 30 scenarios the 'calibrated' and strict prompts returned identical verdicts — the earlier gain was just test-set composition bias.
The root cause: the line between 'sufficient' and 'insufficient' output isn't one-dimensional. It depends on at least three largely independent factors — who consumes the output, what environment it's deployed to, and what breaks if it's wrong. Most real outputs sit in the interior of this 3D boundary surface, where multiple defensible judgments cross. No model, prompt, or voting scheme removes that ambiguity; each just relocates the operating point along the same surface.
The practical takeaway is to design around the wall rather than chase it. Cheap deterministic checks handle obvious garbage, a single calibrated LLM call judges the semantic residual, and split verdicts escalate to a human. The false-positive/false-negative tradeoff is structural — the only design mistake is refusing to pick a side.