Catching the Zero-Token Billing Bug in AI Streaming Responses
stream_billing_gate.py offline tool detects when AI streaming responses deliver full text but the usage frame is lost, leaving billing at zero tokens.
In streaming AI responses, text arrives first and token accounting arrives last in a terminal usage frame. If that final frame is dropped, truncated, or zeroed due to a flaky connection, the user still receives the complete answer, but the client logs zero output tokens — silently, with no exception thrown. The bill ends up reflecting a cost that never happened, even though real tokens were spent.
stream_billing_gate.py addresses this by reading recorded stream logs offline, reconstructing the delivered text, and reconciling it against the logged usage frame. When text was delivered but usage is zero or missing, the tool blocks with a 'delivered-but-unbilled' verdict. Since it has no real tokenizer, it uses word count as a conservative floor — not an exact measurement, but enough to prove the logical claim: if text was delivered, real token usage must be above zero, making a logged zero provably wrong.
The tool defines four verdicts: OK (consistent), UNDERCOUNT (partial loss, warns), BLIND (zero billing, blocks), and EMPTY (nothing delivered). It auto-detects wire formats from OpenAI Chat Completions, Anthropic Messages, OpenAI Responses, and a generic normalized schema, making it usable across different provider logs. For engineers building cost-tracking and observability pipelines on top of streaming APIs, it offers a practical safeguard against silent telemetry loss.