Custom Vulkan Engine Beats llama.cpp by 1.44x for One Model on RDNA3
A hand-written Vulkan inference engine for one model on RDNA3 GPUs decodes 1.44x faster than llama.cpp with token-exact output parity.
A developer built a from-scratch Vulkan inference engine specialized for a single model, Qwen3.6-35B-A3B, targeting AMD RDNA3 GPUs. Rather than forking llama.cpp, the project hand-writes GEMV/GEMM kernels for every GGUF weight format present (Q8_0, Q6_K, IQ4_XS, IQ3_XXS, F16), reaching 90-97% of VRAM bandwidth, and fuses the model's entire hybrid architecture — gated-DeltaNet recurrence interleaved with 256-expert MoE, GQA attention, and GPU-resident argmax sampling — into a single command-buffer submission per decode chunk.
On identical hardware and the same quantized GGUF, the engine decodes at 190.7 tok/s versus llama.cpp's 132.3 tok/s on an RX 7900 XTX (1.44x), and 147.1 vs 109.7 tok/s on an RX 7900 XT (1.34x), benchmarked against a same-day llama.cpp build after the author corrected an earlier comparison that had used a stale three-month-old baseline. Greedy decoding output is verified token-for-token identical to llama.cpp, with batched paths validated bit-identical or argmax-stable against serial references.
The project ships as a safe-Rust server exposing the Anthropic Messages API, allowing tools like Claude Code to connect directly, and includes fast prefix-cache restoration (0.3 ms vs 341 ms for re-prefill). The tradeoff is specialization: the engine only supports RDNA3 GPUs and only this one model architecture, illustrating how much throughput remains on the table when kernels and scheduling are hand-tuned per-model rather than handled by a general-purpose runtime.