LLMs Are Honest in Prose but Hallucinate Under JSON Schemas
Study finds LLMs admit uncertainty in prose but fabricate data under required JSON schemas, with 10 of 13 models hallucinating 100% of the time.
A new independent study finds that language models trained to say "I don't know" reliably do so in prose, but largely lose that behavior under structured output formats. Testing 13 models with inputs engineered so no honest answer exists — such as a viral post with engagement counts but no reply text — models answering in free text correctly flagged missing evidence 98% of the time. The same question forced through a JSON schema with required fields produced fabrication in 10 of 13 models, 100% of the time.
Adding an escape value like "insufficient_evidence" to the schema fixes the problem for frontier models such as GPT-5.5, which take that exit every time. It does almost nothing for open models, which still fabricated 60-100% of the time even with the escape present. Grammar-constrained decoding confirmed the escape token was reachable — models used it, but only on a low-stakes field, never on the three fields that actually constitute the invented answer.
The study also found that prompt-level anti-hallucination instructions, effective in free text, stopped working under required-field schemas for most models tested, and that resistance to fabrication does not scale predictably with model size. The researcher released an open-source linter, phantomfill-lint, to flag schemas missing abstention paths, and published the work as a preprint accepted at the AIMS workshop at COLM 2026.
The takeaway for engineers: published abstention benchmarks measure prose behavior, not the JSON-mode and function-calling interfaces actually used in production. Schema design should be treated as a safety-relevant decision, not just an API contract detail.