Why reformatting prompts beats quantization for CPU LLM inference
Measured on a free ARM CPU box: rewriting prompts as label:value fact sheets cut LLM prefill time 6.5x while quantization tricks hurt accuracy.
An engineer running a CPU-first llama.cpp inference server on a free Oracle ARM box (4 cores, €0/month) set out to speed up prefill — the phase where a model must read the entire prompt before producing a single token, and the dominant cost on constrained hardware.
Three obvious optimizations failed. A year's worth of upstream ARM kernel improvements produced 0% gain. A coarser quantization (Q4_0) sped up prefill by 37% but dropped 5 of 20 answers on a fact-extraction test. Halving active experts in a mixture-of-experts model gave a 44% speedup but silently corrupted the KV cache — a cache built with 4 experts scored 11/20 instead of 14/20 when read back with 8.
What actually worked was rewriting the input text itself. Converting the same page from prose into 'label: value' fact-sheet form cut prefill time 6.5x and, unexpectedly, improved accuracy from 19/20 to 20/20. Attention analysis showed why: in prose the model could see the right value but failed to bind it to the question; the structured format made that binding mechanical rather than semantic, flipping a near coin-flip attention ratio to roughly 7:1.
Two further findings apply broadly to small-model CPU deployments. The tied output-embedding head consumes nearly a third of per-token memory bandwidth scoring vocabulary the deployment never uses, and trimming it to 32k tokens gave a safe +17.8% decode speedup with no accuracy loss. Separately, llama.cpp's weight repacking for NEON kernels trades 4.2 GB of extra RAM for a 12.8% speed gain — a decision that determines whether a model fits at all on an 8 GB device.