Severity, Not Flat Pass-Rate, Should Gate Agent Evals
severity_gate.py is an offline, stdlib-only CI tool that gates agent eval runs by failure severity instead of a flat pass-rate, returning SHIP, REVIEW, or BLOCK.
This post highlights a common blind spot in agent CI pipelines: collapsing eval results into a single average pass-rate. The author argues this masks real risk, since a mean weighs a critical failure—like a customer email leaking into logs—the same as forty trivial whitespace failures. The proposed fix, severity_gate.py, reads a finished eval results JSON along with an optional policy file and returns a SHIP, REVIEW, or BLOCK verdict.
The logic is straightforward: based on a user-assigned severity_class field, a single failure in a blocking class (critical by default) forces a BLOCK regardless of the aggregate pass-rate; a failure in a review class or a below-threshold pass-rate triggers REVIEW; clean runs get SHIP. A demo fixture shows that changing the severity_class of just two failing cases from low to critical flips the verdict from SHIP to BLOCK while the 92.5% pass-rate across 40 cases stays exactly the same.
The author is explicit about limitations: the tool doesn't detect PII, judge correctness, or discover new failures—it only re-weights failures already labeled by humans, so bad labels produce a bad gate. Built with only Python's standard library (json, sys), it runs offline with no network calls and produces deterministic output, using exit codes 0/1/2 for CI integration. Its value for engineers lies in demonstrating concretely why ship decisions should hinge on failure distribution rather than a flattened average.