VetoBench Tests Whether AI Agent Memory Retains Rejected Decisions

Most memory benchmarks measure whether retrieval surfaces the right stored fact (Recall@k, MRR). VetoBench asks a different question: does an agent re-propose something your team already rejected — like Redis after it fell over under load? The benchmark uses 24 synthetic engineering decisions, 10 carrying a structured "rejected" field, paired with tasks designed to invite that exact rejected approach, and scores violations with a deterministic (non-LLM) judge.

With no memory at all, the agent re-proposed a rejected approach in 80-90% of tasks. Simply dumping vetoes into context as flat text dropped that to 0% — proving that storing the rejection matters more than retrieval sophistication at small scale. Testing real Mem0 ([email protected]) with its own extraction and search showed a decent headline violation rate (0-20%), but archived retrieval contexts revealed the veto was missing in 38% (19/50) of cells due to lossy fact-extraction — and violations concentrated there: 26% when the veto was absent versus 3% when present. Summarization pipelines tend to drop negative-space facts like "we rejected X."

Running the identical transcripts through RoBrain's own production extractor told a different story: 100/100 vetoes survived extraction, 50/50 retrieval contexts contained them, and 0/50 violations occurred — because RoBrain's extraction prompt treats rejected options as a first-class output field, not an incidental detail. The author is candid about limits: at this small corpus size, a flat-file dump ties RoBrain, and the retrieval advantage is only claimed at real scale (hundreds of decisions). The benchmark is open source and accepts third-party memory adapters, including ones meant to expose weaknesses.