We gated CI on six LLM eval frameworks — only two survived

Over roughly eight months, six open-source LLM evaluation frameworks were wired into a real GitHub Actions merge queue and run against production pull requests. The goal wasn't ranking feature lists, but testing which tools could reliably gate a merge — pass or fail the same way, every time. The finding: deterministic checks (string match, regex, JSON-schema) return in under a second, cost nothing, and never flake, while LLM-as-judge scores drift near their threshold and can grade an unchanged output differently from one day to the next.

The motivating incident was a judge-based gate with a 0.8 threshold that blocked fourteen PRs and a release over a weekend — same prompt, same model, no fixed seed, score moving from 0.83 to 0.78 on identical input. Incidents like that train teams to force-merge past a gate, and once that habit sets in, the gate stops protecting anything. The evaluation criteria that followed were determinism, speed, cost, wiring effort, and signal quality.

Promptfoo and DeepEval came out ahead. Promptfoo's CLI-based deterministic assertions never flaked and ran in milliseconds, with judge-graded checks kept optional and non-blocking. DeepEval offered a similar split inside a pytest workflow. Tools built around judge-first metrics or script-driven evaluation — RAGAS, Arize Phoenix, Future AGI, MLflow evaluate — remain useful for quality measurement and tracing but proved less suited as hard merge-queue gates.

The practical takeaway for engineering teams: before any eval metric goes into a blocking CI check, verify it returns the same verdict on unchanged input. Judge scores can still be valuable as advisory signals, but the check that blocks a merge should stay deterministic.