SkewAdam: A New Optimizer Reducing MoE State Memory by 97%
SkewAdam reduces MoE state memory by 97%, allowing a 6.7B model to fit on a 40GB GPU.
SkewAdam is a newly developed optimizer aimed at overcoming the significant VRAM bottleneck in Mixture-of-Experts (MoE) training. Traditional optimizers like AdamW consume large amounts of memory, but SkewAdam employs a tiered state allocation based on parameter behavior. This innovation reduces optimizer state memory by 97.4% to just 1.29 GB, enabling a 6.78B MoE to fit comfortably on a single 40GB GPU without compromising convergence or router stability.