Prefill/Decode Disaggregation Can Worsen Tail Latency, Not Fix It

Splitting prefill and decode across separate GPU pools looks architecturally clean: compute-heavy prompt processing and memory-bandwidth-heavy token generation scale independently. But disaggregation doesn't remove the bottleneck—it redistributes it across routing, prefill queues, KV transfer, and decode admission. Each phase can report local success on its own dashboard while the user still waits on end-to-end TTFT.

A July 2026 study on a production-style 2-prefill/2-decode A100 setup found prefill execution itself accounted for only 2-23% of P95 TTFT, with queueing and inter-node KV transfer consuming the rest. Static phase ratios collide with heavy-tailed real-world traffic, stranding decode capacity while prefill queues grow.

Proposed fixes include conditional deflection (letting decode nodes absorb chunked prefill work only when it won't violate TPOT SLOs), progressive KV transfer techniques like Lynx that let speculative decode start before all bytes arrive, and model-dependent compression policies such as SpectrumKV rather than one-size-fits-all quantization. Routers should estimate predicted completion time across the whole path instead of picking the shortest queue, and teams must track both TTFT and TPOT/TBT as separate SLOs.

The takeaway: disaggregation is not a one-time topology choice but a control loop—observe, predict, place or deflect, verify, and adapt ratios continuously. Measuring only within each GPU pool's boundary optimizes the architecture while losing the user.