Solving Moe Load Imbalance in LLM Training via Optimal Transport
TAOT method improves MoE training speed by 43% while reducing communication costs by 74%.
Mixture-of-Experts (MoE) architecture is crucial for scaling large language models (LLMs), but its dynamic routing leads to significant load imbalance during expert-parallel training. Current dynamic-replica methods focus on optimizing load balance but overlook the costs associated with moving expert weights across a multi-node topology, which can negate the benefits of load balancing. The proposed TAOT method addresses this by modeling the overload on busy ranks and the spare capacity on lightly loaded ranks as an optimal transport problem, achieving a 1.43x speedup in MoE training and a 74% reduction in communication costs compared to existing methods.