Why LLM-as-judge scores flip between identical runs
An LLM judge gate flipped between 0.79 and 0.82 on identical inputs. Sampling temperature, model drift and vague rubrics cause the jitter; k-sampling and a noise band fix it.
An engineer's faithfulness gate — a merge check requiring an LLM judge's mean score to clear 0.80 — failed at 0.79 on one run and then passed at 0.82 and exactly 0.80 on identical re-runs with no code changes. The culprit wasn't the code but the judge itself: non-zero sampling temperature, floating model aliases that silently swap snapshots, ambiguous rubric wording, and borderline scores where tiny sampling noise decides the outcome.
The fix isn't bit-level determinism from a hosted model, but shrinking noise enough that a threshold crossing signals a real regression rather than a coin flip. The approach: set temperature to zero and use a seed where supported, pin the exact judge model and rubric version in the cache key, sample k times (e.g. k=5) and average or majority-vote, quantize scores to a coarse grid to kill sub-grid jitter, and version the rubric prompt like code.
The conceptual shift is treating no single judge score as ground truth: score k times, track the mean and spread, and only fail when the mean falls below threshold by more than the measured noise. A shared Python snippet implements this via stable_judge_score and gate functions, with the SystemExit exit code being the actual CI signal. This cut run-to-run swing from roughly 0.03 to under 0.01, making a red result mean something real.
Practical checks recommended: log the judge call's temperature, diff the model string between passing and failing runs, and measure the metric's run-to-run stdev before trusting any gate — if the swing exceeds the threshold's margin, the gate is measuring noise, not regressions.