Colibri lets 744B-parameter GLM-5.2 run on just 25GB of RAM

Colibri is a single-file, dependency-free C inference engine that runs GLM-5.2's 744-billion-parameter Mixture-of-Experts model on consumer hardware with only 25 GB of RAM and no GPU. It works by splitting the model: roughly 9.9 GB of dense layers stay resident in memory, while 21,504 routed experts (~370 GB) live on NVMe and are streamed in only when the router selects them.

The engine combines several optimization techniques, including MLA attention that shrinks the KV-cache 57-fold, DeepSeek Sparse Attention to cut compute complexity, int8-based speculative decoding via a multi-token prediction head, and a learning cache that automatically pins frequently used experts into free RAM over time. On a development machine this yields a modest 0.05-0.1 tokens/second, though throughput scales up notably with faster disks and more CPU cores.

For engineers, this demonstrates a practical path to running frontier-scale open-weight models without expensive GPU clusters, trading raw speed for dramatically lower hardware costs by leaning on NVMe storage and CPU-only inference.