17 PRs a Day, One QA: Automating E2E Failure Triage

At pdf.net, 18 AI-augmented developers merge an average of 17 pull requests a day — 33 on the peak day — watched over by a single QA engineer. Moving e2e testing from release time to every merge on staging caught regressions earlier, but it also multiplied the number of red runs needing quick human triage. The team's answer was a reusable GitHub Action, fired after every e2e run, that feeds failed tests, logs, diffs and the deploy range to Claude (Opus with a Sonnet fallback) and asks for a verdict — likely-pr-related, likely-not-pr-related, or insufficient-data.

The verdict drives routing, not just messaging: PR-related failures open a Linear ticket with a Slack thread, while anything else triggers a single automatic rerun before any ticket is created — a green rerun means it was a flake, a red one escalates with an on-call mention. The piece also walks through the production bugs the team hit and fixed: an idempotent recovery sequence to stop tickets from staying open forever after a partial failure, a pure recovery-cause classifier that stops the bot from crediting innocent PR authors for fixes they didn't make, and a dual-path dispatcher so mixed runs (some tests fixed, others newly broken) don't skip recovery.

Combined with noise-control measures — deduplication, auto-close with recovery notes, channel separation, and manual muting — the system reportedly costs less per month than a single hour of engineering time. For teams scaling PR throughput faster than QA headcount, it's a concrete blueprint for offloading mechanical failure triage to an LLM-driven decision layer rather than another hire.