How a shared JSONL file inflated our eval score to 0.94
A retrieval-based few-shot selector shared its index with the eval set, inflating accuracy from 0.79 to a fake 0.94. Here's the contamination check that catches it.
A ticket-routing eval reported 0.94 accuracy for five weeks. The real cause: a dynamic few-shot selector built its retrieval index from the same labeled_tickets.jsonl the eval set was sampled from. Every eval case retrieved itself as its own nearest neighbor, gold label attached, pasted directly above the question. The model wasn't reasoning — it was copying. Measured against unseen tickets, true accuracy was 0.79.
The broader lesson: contamination isn't only a training-time problem inherited from a vendor. Any pool you draw prompt content from is part of your eval's input surface. If your eval set and few-shot pool share a parent file, the leak lives in your own repo, shipped by your own team.
The fix is structural, not just procedural: split pools by content hash at the source, and audit the actual rendered prompt, not only the training corpus. A two-pass check — normalized hashing for verbatim/cosmetic duplicates, plus n-gram containment scoring (not Jaccard) — catches both exact leaks and cases where a short eval item is buried inside a longer few-shot example.