How RAG Hallucinations Poison Your Vector Database

A fintech RAG pipeline quietly poisoned itself when its extraction LLM guessed an illegible fiscal_year field and embedded that hallucinated value alongside the source text. Nothing looked broken — retrieval stayed fast and the dashboard stayed green — but the answers it served were wrong. Adding a second LLM as a validator didn't help either: instead of catching errors, the two probabilistic models fell into a sycophancy loop, agreeing with each other rather than checking against ground truth. Tightening the prompt with strict instructions made things worse, making the validator overly defensive and driving up API costs.

The actual fix moved validation out of the model and into deterministic code: the fiscal_year must literally appear in the source text via regex, company_entity is cross-checked against a hardcoded SQL table of real entities, and nothing reaches the vector store without first staging in a quarantine table. That single change stopped the data poisoning and roughly halved API costs.

The broader lesson for engineers is that probabilistic systems need deterministic boundaries. LLM-as-a-judge remains useful for offline, batch evaluation, but it should never be the sole integrity gate on a production write path. The pattern mirrors validating writes before they reach agent memory — the model proposes, but code decides what becomes trusted state.