The AI Agent Hunting Bugs in My App While I Sleep
One AI agent hunts bugs in a live app hourly while another reproduces and fixes them, using confidence scoring and reproduction rules to stay trustworthy.
A developer built an AI agent that hourly probes a real, signed-in desktop app using Playwright over the Chrome DevTools Protocol, clicking through routes, opening dialogs, and filing GitHub issues with screenshots whenever something looks off. A second 'fixer' agent picks up those reports, reproduces each bug itself, writes a patch, captures before/after proof, and opens a pull request for human review.
The real challenge wasn't finding bugs but making the agent honest about them. Every finding gets classified as a genuine bug, expected-but-bad UX, an environment issue, a test gap, or inconclusive, each tagged with a confidence level, and only high-confidence, reproducible bugs get filed. This discipline surfaced quiet, easy-to-miss issues, like chats stuck loading forever, silently failing settings fields, and panels collapsing to a single pixel. In one notable loop, the hunter flagged a missing feature, the team built it, the same agent later caught a bug the new feature introduced, and the fixer resolved it.
The system has clear limits: CDP can only see inside the app's web view, missing OS-level dialogs and file pickers, and silent failures can trick the agent into narrating false success, requiring extra engineering to make failures loud. It's also explicitly not a CI gate, since it drives a slow, realistic, live app rather than fast per-commit checks.
The key takeaway for engineers is that the automation mechanics are simple, but enforcing honesty, through classification, confidence scoring, and mandatory reproduction before fixing, is what turns an autonomous bug hunter into something genuinely trustworthy.