How a Benchmark Turned an MCP Security Proxy From 9% to 63%
An open benchmark shows how the mcp-bastion MCP security proxy grew from 9% to 63% attack-surface coverage through iterative testing.
AI agents built on the Model Context Protocol (MCP) now read files, call APIs, and query databases — creating a fast-growing attack surface. To find out how much of that surface security proxies actually cover, one engineer built mcp-defense-bench, an open, vendor-neutral benchmark that pairs every malicious test case with a matched benign twin to eliminate false positives.
What started as a scoring exercise became an iterative development loop: the open-source mcp-bastion proxy climbed from 9% attack-surface coverage to 63% as each measured gap was closed and re-verified. Along the way, the benchmark caught up with freshly published attacks — including ShareLock, a multi-tool exploit invisible to single-tool scanners but detectable through cross-tool correlation only a proxy sitting across multiple servers can perform.
The final push shifted the tool from merely warning to actively blocking: evasion normalization, inline secret redaction, and default-blocking for high-confidence checks — all while holding zero false positives across 35 test cases. Just as importantly, the study is candid about limits: 5 of 24 attack vectors, including supply-chain attacks, sandbox escapes, and consent fatigue, cannot be solved by any content-scanning proxy, underscoring the need for defense-in-depth.