Generating the database from the answer key in text-to-SQL benchmarks

A UIUC audit published for VLDB 2026 found that 52.8% of BIRD Mini-Dev and 62.8% of Spider 2.0-Snow annotations in widely used text-to-SQL benchmarks are simply wrong, as confirmed by human SQL experts. Correcting just 100 examples shifted relative performance of 16 open-source agents by up to 31% and moved leaderboard rankings by as many as 9 places. The root cause is structural: these benchmarks build a database first, then have question writers and annotators infer questions and gold SQL from data they didn't design, introducing semantic mismatches, schema misunderstandings, and domain-knowledge errors.

One developer inverted this pipeline. Instead of deriving answers from data after the fact, the ground truth — say, exact monthly revenue or a specific fraud rate — is declared first as a specification, and a synthetic database is generated to satisfy it exactly. Questions are then derived from the declaration itself, so the expected answer isn't computed by reading rows; it's true because the data was built that way. Generation is a deterministic two-stage process: allocating row counts and drawing values, then rescaling to hit the declared target precisely.

Crucially, the generator doesn't grade its own work. The written CSV files are queried independently by DuckDB, a system sharing no code with the generator, and observed results are checked against declared values. In the published demo pack, 5 of 40 candidate questions were dropped because integer arithmetic (e.g., a 2% rate not landing evenly on a row count) made the exact target unachievable — and this rejection was recorded transparently in the manifest, contrasting sharply with static benchmarks that silently ship wrong answer keys.

The approach also neutralizes contamination concerns: since an 'evalpack' is just a spec plus a seed, changing the seed regenerates an almost entirely different database with the same declared answers. If a model is suspected of having seen the eval data, rotating the seed preserves the benchmark's validity.