CVE-Bench: A Sandbox Benchmark for LLM Agents Fixing Real CVEs

CVE-Bench is an open-source benchmark that measures how well LLM-driven coding agents can patch real-world Python vulnerabilities. Each task runs inside an isolated Docker container: the agent is handed a repository checked out at a vulnerable commit, given file read/search/edit tools, and allowed up to 20 turns before its work is scored against the maintainer's hidden security test suite.

Tasks ship in three prompt variants — full GHSA advisory text, a behavior-only description (diagnose), or just file/function location (locate) — letting researchers isolate how much context a model needs to actually find and fix a flaw. A run counts as solved only if all security tests pass and no existing regression tests break, and a companion validation tool confirms each task's tests correctly fail on vulnerable code and pass on the fix.

The harness supports OpenAI, Anthropic, and Poolside model providers, writes structured JSON results per run, and can generate comparison charts. Released as independent research under the MIT license, it gives engineers a reproducible way to benchmark agentic models on genuine security-patching tasks rather than synthetic bugs.