Shut-down AI prompt firewall startup open-sources model and 13K attacks

A founder spent close to a year building a paid AI guard product that sat in front of an LLM, inviting users to try to break it for prize money. The business never took off, but it left behind a year of motivated, real-world red-teaming against a live detector — now released as open code, model weights, and a dataset.

The pipeline runs in two stages: a regex gate of 119 patterns covering direct-override phrasing across eleven languages, DAN-style jailbreak framing, template-injection markers, and homoglyph/zero-width character stripping, with a decode-then-scan pass that catches base64, ROT13 and leetspeak payloads first. Whatever survives the gate goes to a fine-tuned DeBERTa-v3-large model, exported to ONNX and quantised to INT8. Re-reading his own code, the author found the docs still described a four-label classifier while the exported model actually only emits two logits — a binary classifier — and corrected the documentation rather than leave the mismatch standing.

On the multimodal side, the system pulls text from images via OCR plus EXIF/PNG/XMP metadata (with a lossy re-encode first to defeat steganography and pixel-level attacks), from PDF/DOCX/XLSX/PPTX files, and from audio via ASR, running the same two-stage filter over all of it. The author isn't aware of another open-source detector that scans images, documents and audio for injected payloads together.

The real payoff is the 13,230 hand-written attack strings — no templates, no synthetic generation, just real people trying to beat a live filter. Identifying and payment data were stripped and anonymised before release. Code and weights ship under Apache-2.0, the base model under MIT, and the stated threat model is explicitly limited to inference-time attacks, not training-time or weight-level compromise.