A 24-Test Readiness Checklist for Deploying AI Agents Safely

A capable model does not guarantee a safe system. The same agent can send duplicate payments, cross tenant boundaries, follow instructions embedded in a retrieved document, or keep executing after a user cancels. Production readiness, this framework argues, is not a property of the model but evidence that the entire pipeline—from input to real-world side effect—stays within an agreed risk tolerance.

The proposed review organizes 24 concrete tests across six gates: identity and authority, tool and side-effect safety, bounded execution and recovery, injection and isolation, data/memory/evaluation, and observability and operations. Each test demands hard evidence rather than assurances—least-privilege tool access, idempotent retries, crash-and-resume testing, treating untrusted content as data rather than instructions, reconstructable traces, and a working kill switch.

Rather than applying one bar to every agent, the framework classifies tasks into five risk tiers based on the authority they hold, from text-only generation to irreversible external effects, with evidence requirements scaling accordingly. Results should be recorded as versioned, expiring artifacts and mapped against structures like NIST's AI Risk Management Framework and OWASP's agentic security threat model. The goal is to replace binary approve/reject decisions with graded outcomes like PASS and CONDITIONAL, making agent safety a continuously re-verified state rather than a one-time sign-off.