When Fine-Tuning Stalls: The Architecture That Took Recall to 92%

A healthcare claims platform was stuck at 45% recall on CPT/ICD medical coding. The team first tried the usual playbook—better prompts, more examples, stronger retrieval—which pushed recall into the low 60s before progress flatlined entirely. The root cause wasn't a knowledge gap in the model but the fact that a large share of coding decisions weren't judgment calls at all: they were already solved, published, deterministic rules from CMS's NCCI edit set. The model was guessing at answers that already existed as a lookup table.

The fix wasn't more model tuning but building a deterministic validation layer alongside it. The team loaded the full 1.73-million-pair NCCI edit set so the model could propose candidate codes while every pair was checked against the table before shipping—only pairs that cleared the rules went through. Production issues followed, from a slow linear scan needing to become an indexed lookup, to handling CMS's quarterly rule updates with scheduled refreshes and version-stamped verdicts. The result: CPT recall rose to 92%, while ICD landed at 76.8% because ICD coding relies more on genuine clinical judgment than fixed rules—a gap that reflects the ratio of rule-based versus judgment-based problems, not model weakness.

The broader lesson is that a model's accuracy ceiling is often architectural, not a training problem. When a metric stalls, the right question isn't 'do we need a bigger model' but 'is this a guessing problem, or does a correct answer already exist somewhere that we're asking a probabilistic system to guess at instead.' For engineers, this argues for treating probabilistic models and deterministic rule systems as distinct components with distinct jobs, rather than endlessly fine-tuning a single system to do both.