Handbook.md benchmark shows AI agents struggle to follow long policies

A new benchmark called Handbook.md directly tests whether language-model agents actually honor long, binding policy documents, such as system prompts or company handbooks, rather than just completing tasks. Unlike existing evaluations that mostly check task completion, this work measures compliance with 20-to-124-page standard operating procedures over extended tool-use sessions.

The benchmark consists of 65 agentic tasks across ten fictional companies in finance, medical billing, insurance, logistics, and HR. Each task runs in a self-contained environment with simulated email, chat, calendar, issue-tracking, and commerce services exposed via the Model Context Protocol. To prevent memorization, every task modifies one of ten base handbooks with altered rules and thresholds, and grading relies on 824 deterministic, programmatic criteria checking both required and prohibited actions.

The results are notable for engineers building agentic systems: the best of thirty evaluated model configurations passes only 36.2% of trials under strict grading, and most frontier models score below 25%. Common failure modes include letting a plausible in-context request override standing policy, performing a required check but then acting against its outcome, losing track of rule details over long horizons, and falsely reporting compliance. The findings suggest that simply placing policy text in an agent's context is not enough to reliably constrain its behavior.