VetoBench Tests Whether AI Agents Forget Rejected Decisions

VetoBench is an open-source benchmark that reframes how AI memory systems should be evaluated: instead of asking whether the right fact was retrieved, it asks whether an agent re-proposes an approach a team has already rejected. Built on 24 synthetic engineering decisions, it runs four memory conditions — none, plain conventions, a flat dump of decisions including vetoes, and RoBrain's ranked retrieval — against the same set of scenarios.

The results show a stark pattern: an agent with no memory re-proposes a previously rejected approach in 80-90% of tasks, while feeding vetoes into context — whether as a flat dump or via RoBrain's retrieval — drops that violation rate to zero. Testing a third-party system, Mem0, surfaces a different failure mode: 38% of recorded vetoes are lost during its extraction pipeline, and violation rates are roughly eight times higher in cells where the veto didn't survive versus where it did.

The benchmark also fairly stress-tests RoBrain's own end-to-end extraction pipeline (robrain-e2e), where all 100 vetoes survived extraction intact. Every run is archived with full contexts and agent replies for reproducibility.

For engineers, the takeaway is that retrieval accuracy alone isn't enough to evaluate AI memory — the costlier failure is reopening a decision the team already closed. VetoBench offers a small, honest, reproducible way to measure exactly that blind spot.