ALIBI: Adversarial Comments Bypass LLM Vulnerability Detectors
ALIBI framework shows LLM-based vulnerability detectors can be bypassed over 90% of the time using adversarial source-code comments.
New research identifies a previously underexplored attack surface in LLM-based vulnerability detection: adversarial source-code comments that manipulate a model's reasoning without altering actual program behavior. The study frames this through a coding agent that introduces deliberate vulnerabilities while strategically inserting misleading comments to evade detection.
The authors present ALIBI, an automated black-box attack framework that iteratively generates and refines adversarial comments using feedback from the target detector's own reasoning. Tested against four representative LLM-based vulnerability detectors—ranging from specialized open-weight reasoning models to frontier multi-agent systems—the attack achieves success rates above 90% across 125 real-world null-pointer dereference cases derived from actual vulnerability-fixing commits, hitting 100% on one system. Comments that steer detector reasoning or fabricate fake tool outputs prove especially effective, and iterative refinement further boosts success rates.
The findings show that all evaluated detectors are highly susceptible, and that simple prompt-level defenses offer only limited protection. Architectural isolation and sanitizing comments before they reach the detector meaningfully improve robustness. The results highlight a fundamental design flaw in current LLM-based security tooling and call for systems that carefully calibrate trust between natural-language context and verifiable program evidence.
This synthesis was produced from its source by AI; there is no human editor or manual review step. How we work