SciCode-Verified: Flawed Benchmark Masked True LLM Coding Skill
Fixing 263 defects in the SciCode benchmark lifted LLM scientific-coding accuracy from 60% to 98%, revealing the test—not the models—was flawed.
SciCode has become a standard measure of scientific-coding ability in language models, feeding into the Artificial Analysis Intelligence Index and government evaluation suites. Yet the strongest 2026 models had plateaued around 60% subproblem accuracy, with newer models failing to surpass their predecessors. A new domain-expert audit finds the stagnation traces back to the benchmark itself, not model capability.
Researchers manually reviewed all 65 SciCode test problems and uncovered 263 defects, 192 of which—spread across 91% of the main problems—caused correct, instruction-following solutions to be wrongly rejected. Causes included non-reproducible gold answers, overly tight tolerances, and self-contradictory specifications. Notably, 78% of these score-suppressing defects required specialized physics or mathematics expertise to detect, not simple proofreading.
After correcting every confirmable defect to produce SciCode-Verified, the team re-ran twelve frontier models on the fixed benchmark. Subproblem accuracy jumped from 45-60% to 84-98%, and main-problem accuracy rose from 9-27% to 69-92%. The findings suggest that when evaluating model capability, engineers should scrutinize the benchmark's construction as closely as the model itself—flawed test instruments can drastically understate real performance.