SDABench: A New Benchmark Testing LLMs on Scientific Discovery
SDABench evaluates LLMs on six scientific capabilities beyond code execution, exposing major gaps in assumption selection and mechanistic reasoning.
SDABench is a new benchmark that evaluates large language models on scientific data analysis through six distinct capabilities—descriptive, exploratory, inferential, predictive, causal, and mechanistic—rather than just code execution or workflow completion. It spans five domains (Biology, Chemistry, Environment, Geography, Physics) with 527 real-data instances and 6,000 synthetic instances in both multiple-choice and open-ended formats.
Testing 15 representative LLMs revealed strong performance on descriptive tasks but sharp degradation on assumption selection, latent-process modeling, and mechanistic reasoning. A five-stage error analysis framework shows that models can identify problem scope reasonably well but struggle with choosing correct analytical procedures, modeling variable relationships, and drawing valid scientific conclusions.
For developers building scientific AI tools, SDABench offers concrete targets: improving assumption awareness, methodological selection, and causal/mechanistic inference. The results suggest current LLMs are far from functioning as reliable AI scientists, despite competence in basic data summarization.