Why Your LLM-as-Judge Gate Flips Between Runs
Same code, same input, different LLM judge score: why CI gates flap between runs, and how temperature, model pinning, k-sampling and quantization fix it.
An engineer discovers that a merge-blocking faithfulness gate—backed by an LLM-as-judge—fails on one run and passes on the next with zero code changes. This flapping erodes trust more than a real regression would, since people learn to just re-run until the check turns green, letting real regressions slip through on a lucky pull.
Four sources of jitter are identified: non-zero sampling temperature (the biggest and most overlooked culprit), floating model aliases that silently drift to new snapshots overnight, ambiguous rubrics that the model resolves differently each call, and borderline cases where tiny sampling noise decides the tie right where thresholds tend to sit.
The fix isn't perfect determinism but shrinking noise enough that a threshold crossing means something real: set temperature to zero, pin the exact dated model snapshot and rubric version in the cache key, sample the judge k times and average or vote, and quantize scores to a coarse grid to kill sub-grid noise. The real conceptual shift is treating a single judged score as a sample rather than ground truth—computing mean and standard deviation across k calls, and only failing when the mean drops below threshold by more than the measured noise.
Applying this cut run-to-run variance on the metric from about 0.03 to under 0.01, making a red result finally mean a genuine regression rather than noise. The takeaway for engineers: never trust an LLM-as-judge gate in CI without first measuring its run-to-run variance.