The Real Cost of LLM Inference Is Memory Bandwidth
LLM serving costs are driven by memory bandwidth, not FLOPs. Why KV cache movement, prefill/decode disaggregation, and batching define real inference economics.
Discussions about LLM inference cost tend to fixate on FLOPs and token pricing, but the actual bottleneck in production serving is memory bandwidth — specifically the cost of moving the KV cache during every autoregressive decode step. Unlike compute-bound prefill, decode requires reading the entire accumulated KV cache plus model weights to produce a single output vector, making it a low arithmetic-intensity, bandwidth-bound workload where faster chips barely help.
Context length compounds this problem, since the KV cache grows linearly with sequence length and concurrent requests. Every token of retained context becomes a recurring bandwidth cost for the rest of that session, meaning long-context features function less like a capability upgrade and more like a standing infrastructure liability that can quietly inflate cost-per-token.
The major architectural response has been disaggregating prefill and decode onto separate hardware pools, since the two phases have opposing resource needs — one wants raw compute, the other wants bandwidth and cache locality. This separation allows decode requests to be batched to amortize fixed memory-read costs, though it introduces added complexity in request routing and cache transfer between pools.
Practically, this reframes several engineering decisions: quantization becomes a decode-cost lever as much as an accuracy tradeoff, batch size becomes the primary way to shift a bandwidth-bound workload toward compute-bound efficiency, and aggressive context retention should be budgeted as an ongoing infrastructure cost rather than a free feature. Vendor comparisons based on peak FLOPs largely fail to predict real costs for decode-heavy production traffic — memory bandwidth, cache capacity, and interconnect speed are the specs that actually matter.