MoE Expert Streaming Runs a 120B-Parameter LLM on an Android Phone
An open-source Android app streams MoE experts from flash storage to run a 120B-parameter LLM on phone CPUs, with 30B models at usable speeds.
A developer got gpt-oss-120b (Q4_K_M, 60GB on disk) running on a OnePlus 15R using nothing but four CPU cores — no GPU, no NPU — at 1.3-2.2 tok/s. While that's mostly a demonstration, the same approach makes Qwen3-30B (5.2 tok/s) and Gemma-4-26B (4.1 tok/s) run at genuinely usable speeds on the same hardware.
The technique exploits how MoE models work: only a handful of experts activate per token (gpt-oss picks 4 of 128). Shared weights stay resident in RAM, while the experts a given token needs are read off flash storage with O_DIRECT right before their layer executes, overlapped with compute. Plain mmap of the same file manages only 0.089 tok/s, so the streaming scheme delivers roughly a 14x speedup. A CI test verifies correctness by comparing streamed generation against fully-resident generation token by token, failing on any divergence.
The hardest engineering problem wasn't the streaming itself but stopping Android from reclaiming resident weights mid-generation — most of the effort went there. The project wraps vanilla llama.cpp as a submodule with no fork and uses only public APIs; qwen3moe, qwen2moe, gemma4 and gpt-oss are supported today, and adding a new architecture is a one-line change. It's released under Apache-2.0 with a prebuilt APK on GitHub.