How Airbnb Cut LLM Evaluation Time From Weeks to a Day
Airbnb's four-layer LLM evaluation stack uses deterministic caching and micro LoRA adapters to cut iteration time from weeks to a single day.
Shipping production LLM systems means iterating on something inherently non-deterministic: judges score identical outputs differently across runs, and generated references regenerate as different strings. Airbnb engineers built a four-layer infrastructure stack to make evaluation trustworthy enough to tell whether a small score change reflects real improvement or measurement noise.
The core insight is separating epistemic uncertainty (model or judge limits) from aleatoric uncertainty (task ambiguity), since conflating the two leads to errors like misclassifying valid high-entropy answers as hallucinations. A per-sample cache, keyed by input and configuration for both references and judge scores, makes evaluation deterministic and resumable, eliminating most of the noise introduced by the testing process itself rather than the model.
For model changes, instead of full adapter retraining—which takes days and risks regressing working behavior—the team uses small 'micro adapters' with rank under 50, layered on a frozen base model. These train in under an hour on a single GPU and ship like software hotfixes, gated against regressions and canary-deployed with rollback. The result is a same-day loop for diagnosing, fixing, and validating issues, replacing a multi-week retraining cycle.