LLM Inference Latency: Why Memory Bandwidth Beats FLOPS
Why LLM inference speed depends on GPU memory bandwidth, not FLOPS. Real tok/s benchmarks across GPUs, quantization levels, and batch sizes.
An H100 offers roughly 15x more raw FP16 compute than a T4, yet running a 7B parameter model on it is closer to 150x faster. The explanation lies not in TFLOPS but in memory bandwidth: autoregressive decoding requires streaming every model weight from VRAM for each generated token, leaving compute units idle most of the time.
This piece breaks down the math behind prefill (compute-bound) versus decode (bandwidth-bound) phases, and derives theoretical tok/s figures across a dozen real GPU and model combinations — from T4 to B200, FP16 to INT4. Quantization emerges as the primary lever for reducing bytes moved per token, delivering 3-4.5x speedups without new hardware, though it does little for time-to-first-token. Continuous batching and tensor-parallel communication overhead round out the picture of what actually determines throughput in production.
For engineers sizing GPU fleets or choosing quantization strategies, the takeaway is concrete: inference throughput is best modeled as bandwidth divided by bytes-per-token, not as a function of advertised compute specs.