Building an AI that pentests your AI, with proof required

Ask an LLM to pentest its own system and you typically get a report full of hedge words — 'this endpoint may be vulnerable' — with no way to separate a real finding from a hallucinated one. The author tackled this by building agent-redteam, a local adversarial harness that attacks a real production LangGraph-based copilot and treats Anthropic's agent threat taxonomy not as a list to admire but as an executable test plan.

The key fix is giving every attack class a deterministic, LLM-free 'oracle': a small function that answers one yes/no question, like whether a canary string planted in one tenant's document leaks into another tenant's response, or whether a forged token returns a 2xx status instead of 401/403. The LLM's role is confined to generating creative attack payloads — role-play, encoding tricks, indirect injection — while the pass/fail verdict never passes through a language model.

The five original threat categories were rewritten as numbered playbook files, each pairing a target and technique with exactly one named oracle. Confirmed exploits get added to a regression corpus so fixed vulnerabilities stay fixed. In the author's most recent run, only one of 31 attacks was confirmed, illustrating how much noise this approach strips away compared to typical LLM self-assessment.

The broader lesson: let the model be the attacker, never the judge. Routing verdicts through deterministic checks rather than another LLM call is what turns AI-assisted security testing from plausible-sounding prose into genuinely reproducible proof.