ZML/LLMD Alpha: One LLM Server Across CUDA, ROCm, TPU, Metal
ZML/LLMD alpha runs LLaMa, Gemma, Qwen and Mistral models across NVIDIA, AMD, TPU, Intel and Apple Metal in one server, with DFlash speeding up inference.
ZML/LLMD is a self-contained LLM inference server built on the Zig and MLIR-based ZML framework, capable of running the same codebase across NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI and Apple Metal. The alpha release supports modern serving features—continuous batching, paged attention, tensor-parallel sharding, prefix caching and tool calling—uniformly on every backend, alongside Prometheus metrics.
A built-in virtual filesystem lets models be streamed zero-copy directly from HuggingFace, S3 or GCS without pre-downloading. A new speculative decoding technique called DFlash, launched with Gemma 4, can boost per-user token throughput up to 10x on supported models and works transparently across hardware platforms.
Docker images are heavily optimized per platform with smart layer ordering and compression, cutting pull times dramatically, while the CUDA image includes an automatic driver compatibility layer. Because the ahead-of-time compilation model avoids hidden runtime compilation, latencies stay flat and predictable; published benchmarks on H100, MI300X, Intel B70 and TPU v6e show competitive throughput and latency.
For engineers, this means a single server image that behaves consistently across heterogeneous accelerator fleets, with fast cold starts and reduced operational overhead—particularly useful for teams managing mixed GPU/TPU infrastructure.