Text-to-SQL Fails Because of Your Warehouse, Not the Model
Text-to-SQL accuracy collapses on real enterprise warehouses despite strong benchmark scores. Evidence points to undocumented schemas, not model quality, as the cause.
Systems that hit 91% accuracy on the academic Spider benchmark collapse against real enterprise warehouses in Spider 2.0, where GPT-4o's score drops from 86.6% to 10.1%. A failure analysis on the BIRD benchmark tells the same story: over 80% of errors trace back to the model misreading schema meaning or table structure, not writing bad SQL syntax.
Holding the model constant while changing only the context makes the pattern clear. Pinterest's internal text-to-SQL system raised its correct-table hit rate from about 40% to 90% simply by adding table documentation to search, with no model change. BIRD's optional one-sentence context note lifts both model accuracy and human analyst accuracy by roughly the same 20 points, evidence that the bottleneck is missing information, not intelligence. Even vendor benchmarks concede this: Snowflake reports raw GPT-4o at 51%, jumping past 90% only once a human-authored semantic model is added.
Newer agentic systems that probe the schema at query time have pushed Spider 2.0 scores back up toward 90%, but that's the same fix applied expensively, per query, rather than a replacement for it. An agent can discover that a duplicate table exists or what values a column holds, but it can't discover which of several competing business definitions is the canonical one, because that fact was never written down anywhere it can reach. The fix isn't a better model, it's modeling the data, documenting definitions, and marking what's canonical.