Bw24: From-Scratch Rust+CUDA LLM Engine Tuned for RTX 50 Blackwell

Bw24 is a from-scratch LLM inference engine written in Rust and CUDA, purpose-built for a single target: RTX 50-series Blackwell GPUs (sm_120a), starting with an RTX 5090 Laptop's 24 GB VRAM. It skips existing frameworks like ggml or llama.cpp entirely, with every kernel hand-tuned against measured hardware limits and benchmarked directly against llama.cpp on the same machine.

The standout feature is MTP (multi-token prediction) speculative decoding, which delivers up to 2.3x throughput over llama.cpp's best speculative configuration across supported Qwen models, while gating every speculative output to be token-identical to plain decode — so speed gains never alter what the model produces. The project ships trimmed drafter heads, an OpenAI-compatible server, and support for six models spanning Qwen3.5/3.6 and several Gemma-4 sizes, plus early support for expert-offloading schemes (Hy3, MiniMax-M3) that spill weights across VRAM, RAM, and NVMe.

For engineers, bw24 shows what single-target, exactness-first kernel engineering can extract from one specific GPU generation: an FP8 KV cache, occupancy-tuned attention tiles, and FR-Spec draft trimming (150 MB down to 18 MB with near-unchanged acceptance) all deliver measurable gains without loosening correctness guarantees. It's a narrow but rigorously benchmarked alternative to general-purpose engines — suited to single-user, single-model serving on RTX 50-series hardware, not a replacement for multi-GPU or batched-serving stacks like vLLM or SGLang.