WISP: A CUDA Engine for Streaming 744B+ Parameter MoE Models on Consumer Hardware
WISP is a CUDA engine for streaming 744B+ parameter MoE models on consumer hardware.
WISP is a CUDA engine designed for streaming 744B+ parameter MoE models on consumer hardware. Instead of loading the entire model into RAM, it streams only the necessary experts, enhancing memory efficiency. By integrating VRAM, RAM, and NVMe into a single memory hierarchy, WISP optimizes MoE inference.