LLM Latency Budgets: Speed Up AI Workflows Without Guesswork

AI features rarely slow down all at once. Delay creeps in through longer prompts, bigger retrieval results, extra tool calls, and unmeasured retry paths until users notice before dashboards do. This piece proposes a stage-level latency budget model as an alternative to vague 'make it faster' goals — defining time allowances, degrade behavior, and quality floors for every step of an AI workflow.

The framework separates workflows into classes (inline assist, interactive answer, tool workflow, background agent) with distinct latency expectations, then walks through a sample stage budget table covering retrieval, time-to-first-token, tool calls, and validation, plus a TypeScript pattern for enforcing budgets at runtime with timeouts and fallbacks.

For engineers, the core insight is that end-to-end latency only shows that a user waited — it doesn't reveal what stole the time. Stage-level tracing, budget-exceed rates, and degraded-response metrics turn latency from a guessing game into an intentional, enforceable contract.