An alternative to LLM judges: risk-based deterministic routing
Instead of using an LLM to judge quality, this four-layer design routes tasks by risk using deterministic code, diff review, and fixed sampling.
Building on three prior articles that dismantled popular 'agent quality gate' designs through six reproducible experiments, the author argues every tested approach shared one flawed assumption: that an LLM can semantically judge whether another LLM's output is correct. Rather than proposing a smarter judge, this piece offers a concrete alternative — stop judging correctness, judge risk instead, using only deterministic code with zero new LLM dependencies.
The proposed four-layer architecture routes tasks by output type before they reach an agent pipeline: verifiable outputs (code, JSON) pass through automatic compile/schema checks; high-risk tasks (money, legal, privacy) go straight to humans; low-risk content auto-releases; medium-risk content gets a diff review that highlights only what changed, replacing full-text quality judgment with a narrower, lower-cognitive-load question. A statistical process control layer flags behavioral anomalies — execution steps, output length, character ratios — instead of relying on semantic clustering, and fixed-rate sampling (e.g. 5% for zero-shot generation) replaces confidence scores that would otherwise require a costly feedback loop.
The author subjects the design to the same critical scrutiny used earlier, openly noting its limits: task classification isn't free and risks mislabeling by cautious owners, diff review only applies when a prior document version exists, 5% sampling still misses roughly a third of defects on any given day if the true defect rate is high, and intercepting sensitive tool calls carries its own inference cost. The result is presented not as a perfect solution but as a pragmatic, fully deterministic control layer with clearly quantified trade-offs.