If 30% of SWE-Bench Pro Tasks Are Broken, Add an Uncertainty Budget
OpenAI's SWE-Bench Pro audit found ~30% of tasks broken. Learn how to version task validity and report uncertainty intervals in benchmarks.
An OpenAI audit published July 8, 2026 found that roughly 30% of SWE-Bench Pro tasks are broken, undermining the common leaderboard assumption that every task in the denominator is valid and equally interpretable.
Rather than discarding benchmarks, the recommended fix is to version task validity, retain disputed cases, and publish how conclusions shift across different denominators. Model result (pass/fail/infrastructure_error) should be tracked separately from task validity (valid/broken/disputed), with each row carrying provenance for dataset, harness, and model-config revisions.
The piece proposes reporting three numbers — a valid-only score, an all-attempted score, and a validity-sensitivity interval — plus an acceptance rule requiring model comparisons to share the same eligible task set before declaring a winner. The architectural takeaway: a benchmark task isn't just an array entry; it's a versioned claim that input, oracle, environment, and scoring path can actually separate capability from noise.