CTA-Pipelining: A Latency-Oriented Scaling Method for Multi-GPU Systems
CTA-pipelining enhances performance in multi-GPU systems by focusing on latency-oriented scaling.
The evolution of compute infrastructure has transformed multi-GPU systems into tightly integrated shared-memory structures. However, current software still treats these systems primarily as high-speed networks. The demand for serving Large Language Models under latency constraints has shifted GPU workload optimization from throughput-driven to latency-bound, leading to the introduction of CTA-pipelining. This execution paradigm exploits dependencies at the Cooperative Thread Array level, enabling concurrent execution of dependent kernels across GPUs. Results indicate that CTA-pipelining can reduce latency by up to 31.8% compared to micro-batching.