397B MoE Model Runs on a Single RTX PRO 6000 96GB GPU

Krasis, an MoE-focused inference runtime, demonstrated interactive execution of the 397-billion-parameter Ornith-1.0 model on a single NVIDIA RTX PRO 6000 Blackwell 96GB workstation GPU. Since the model can't fit entirely in VRAM, Krasis keeps expert layers resident in CPU RAM and dynamically streams roughly 43% of routed experts into VRAM as needed.

The run uses INT4-quantized experts, HQQ4 attention, and 4-bit KV caching, with peak process memory around 202GB — meaning a 256GB DDR5 system is sufficient to host the model. Measured throughput reached 2,354 tok/s prefill at ~40,000 tokens and a sustained 20.4 tok/s decode over 250 tokens. Krasis's Adaptive Cold Mass Pruning feature, which skips low-ranked non-resident expert routes while omitting only ~1.8% of routed probability mass on average, boosted decode speed by 9.1%.

The same runtime scales down efficiently too: smaller MoE models that fit entirely in system RAM run much faster (around 117 tok/s decode for 35B-class models on an RTX 5090), and Ornith-397B itself can even run on a single 32GB RTX 5090 at roughly 7.9 tok/s for patient users. The results are a notable data point for running very large MoE models locally on consumer and workstation-grade hardware without multi-GPU clusters.