Why Does Your LLM Bill Grow Even When Traffic Doesn't?
LLM costs can rise even when user traffic stays flat. Learn why tracking tokens per operation, not raw requests, reveals hidden cost drivers in production.
An engineer noticed their LLM bill kept climbing even though user traffic remained flat, ruling out obvious causes like pricing changes or accidental model upgrades. The real issue was a mismatch between request-based monitoring and token-based billing: two requests to the same endpoint could carry vastly different costs depending on system prompts, conversation history, tool definitions, and retries, yet dashboards counted them identically.
The fix involved logging each model call tied to the actual unit of work—an operation ID, workflow, model, attempt number, and detailed token counts (input, output, cached, reasoning). This exposed three hidden cost drivers: conversation history silently growing with every turn, tool definitions being resent repeatedly regardless of relevance, and retries or fallbacks appearing as unrelated requests instead of one user action. Verbose model outputs exceeding what the product actually consumed added further waste.
The resulting approach shifts focus away from total spend toward ratios like tokens per operation, model calls per operation, and cached-to-total token ratio, broken down by workflow and model version. This makes distinct cost problems—growing context versus retry loops versus verbose outputs—visibly different, highlighting that cost observability for LLM systems needs its own discipline beyond traditional request and latency metrics.