Evaluating Agentic Autofix: Canary Gates and Failure Taxonomies

GitHub opened its agentic autofix feature for code scanning alerts to public preview on July 10, 2026, per the GitHub Changelog. The announcement raises a harder question: how do engineering teams responsibly measure whether AI-generated security patches are actually trustworthy? The source material isn't a benchmark of GitHub's tool—it's a proposed evaluation methodology for this class of system.

The core critique is that 'percentage of alerts with a generated patch' is a misleading success metric, since patch generation is only the first step in a longer chain: alert, candidate, build, tests, security oracle, human review, merge, and post-merge observation. The proposal argues for running such tools as a bounded canary across repositories with active maintainers, deterministic builds, reversible releases, and no automatic production deployment—logging every attempt, including abstentions and rejections.

Each outcome should be assigned to one of twelve failure classes (scope violation, build failure, regression, semantic-incomplete fix, test manipulation, post-merge escape, and others), preventing a single bad attempt from contaminating multiple metrics. Security oracles matter more than existing unit tests here, since they must specifically fail against the unpatched baseline to prove they can detect the vulnerability at all.

Wider rollout should only follow predeclared gates—zero critical scope violations, zero automatic merges, a minimum completed sample—with results segmented and reported as confidence intervals rather than single averages. Hard stop conditions include privilege escalation, weakened security policy, or secret exposure, triggering immediate rollback and evidence preservation.