7 lessons from building a hard spending cap for LLM APIs
Why LLM API bills silently explode: retry storms, inconsistent token reporting, streaming blind spots, and race conditions in spending caps.
A developer's LLM bill spiked 40x overnight not from a viral traffic surge, but from a retry policy quietly re-sending the same long prompt against a flaky endpoint. Because nothing technically failed, no alert fired. That incident led to building a hard spending cap for LLM calls, and most of the lessons weren't about caps at all — they were about the boring ways cost leaks when the only meter lives on the provider's side.
Key findings include that provider billing dashboards are inherently rear-view mirrors, that per-feature call tagging (not total spend) is the real diagnostic tool, and that every provider — OpenAI, Anthropic, Gemini, Bedrock, Cohere — reports token usage in incompatible shapes. Reasoning tokens are especially treacherous: some providers count them inside the output total, others report them separately, silently undercounting cost for reasoning-heavy models.
Streaming responses hide usage data until the final chunk, leaving much of modern LLM traffic effectively unmetered in naive implementations. Simple check-after-the-call spending caps also fail under concurrency, letting parallel requests blow past limits; the fix is an atomic reserve-then-settle pattern. The single costliest failure mode turned out to be retry storms, not runaway prompts. The findings were packaged into an MIT-licensed, zero-dependency library called budget-guard with adapters for common LLM frameworks.