Why your eval set is too small to catch real regressions

A team celebrated moving their eval pass rate from 87.5% to 90% and logged it as 'prompt v3: +2pp' — except the eval set had only 40 examples, meaning that 2-point gain was literally one test case flipping from fail to pass. That's the default failure mode of eval-driven development: teams see a gap between two numbers and assume it's signal, without checking whether it's just noise.

The piece centers on statistical power — the probability that a test detects a real effect of a given size. Using the two-proportion normal approximation, catching a 5-point drop from 0.90 to 0.85 pass rate at 80% power requires roughly 253 examples. A 50-example set only has about 35% power for that same regression, missing it roughly two times out of three; even 100 examples only gets you to a coin-flip 51%. The article includes a standalone Python script (no dependencies) so engineers can compute this for their own baselines.

Since collecting more labeled data is often expensive, the author offers three practical fixes: report a confidence interval (specifically the Wilson score interval, which behaves better than the Wald interval near the 0.85–0.97 range where most eval scores sit) instead of a bare point delta; replace vague graded rubrics with per-criterion binary labels, which are easier to label consistently and yield more outcomes per example even though correlated criteria aren't fully independent; and fix the effect size you care about before running the eval, not after seeing the result. Paired designs can also use McNemar's test, which needs fewer examples by canceling out item-level difficulty variance.

The takeaway for engineers: size your eval set to the regression you actually need to catch, always pair pass-rate numbers with a confidence interval, and treat small, noisy eval deltas in changelogs with real skepticism.