Why PDF Ingestion, Not the LLM, Broke a Banking RAG Chatbot

After the classifier and semantic cache for a production banking chatbot were working smoothly, the real bottleneck turned out to be something mundane: loading loan policy PDFs into the knowledge base. A naive PDF-to-text extractor flattened tables into meaningless run-on text, fused headers directly into paragraphs, and scattered rate figures with no column context — output that wasn't just wrong, but confidently wrong.

Fixing it required moving to a layout-aware document-intelligence pipeline that detected table boundaries and explicitly stitched tables split across page breaks. Scanned circulars needed an OCR pass with de-skewing, and low-confidence OCR lines were filtered out entirely rather than trusted, since a small gap in coverage was safer than a hallucinated number.

Fixed-size chunking turned out to slice sentences in half, causing the LLM to answer from incomplete context; a hybrid strategy — splitting by heading and paragraph first, falling back to size limits with overlap — fixed retrieval quality. Finally, discovering that multiple versions of the same policy document were circulating exposed a deeper gap: without version and date metadata, the retriever could confidently surface a rate that had been superseded months earlier.

The takeaway for engineers building retrieval-augmented systems: model quality is rarely the limiting factor. Layout-aware extraction, table integrity, OCR confidence filtering, sentence-aware chunking, and document versioning are what determine correctness in high-stakes domains like banking.