OpenAI Retracts Its Own SWE-Bench Pro Recommendation

OpenAI's own audit of SWE-Bench Pro — the benchmark it recommended after retiring SWE-Bench Verified — found that roughly 30% of its 731 tasks are broken, ranging from 200 to 249 depending on the review method. Combining AI investigator agents with human engineers, the audit identified four recurring failure patterns: overly strict tests demanding unstated implementation details, underspecified prompts missing requirements only hidden tests know about, low-coverage tests that let incomplete fixes pass, and prompts that mislead models toward the wrong behavior. The root cause traces back to how these benchmarks are built: real GitHub pull requests, shaped by human collaboration, don't naturally produce clean tasks suited for measuring model capability.

Notably, OpenAI used Codex-based AI agents to conduct much of this audit — inspecting repo history, running tests, and analyzing failures at scale. This marks a new feedback loop where increasingly capable models are used to scrutinize the very benchmarks meant to measure their own progress.

For engineers, the takeaway is practical: treat SWE-Bench Pro scores with a ~30% caveat, since apparent gains may reflect benchmark noise rather than real capability improvements. Anyone building custom evals should source tasks from developers designing tests specifically for models, not repurposed PRs. OpenAI has called for benchmarks built by experienced developers for capability testing, though it hasn't announced what it will use internally next. This episode reinforces an 18-month pattern of coding benchmarks failing quality scrutiny as the replacement cycle keeps accelerating.