Speculative Decoding: The Free Speedup Most Local LLM Setups Skip

A setting available in LM Studio and llama.cpp can make a local model generate 1.5 to 2.5 times faster without altering its output at all. A small draft model guesses upcoming tokens, the main model verifies them in one parallel pass, and every accepted guess is instant while rejections fall back to the big model's own choice. Because the verification uses a rejection-sampling rule that mathematically preserves the main model's output distribution, the result is lossless — only extra VRAM and compute are spent, which is why the technique pays off precisely where GPUs sit idle: single-user local inference.

In 2026 this went from a power-user trick to something shipped by default. Multi-token prediction (MTP) lets model makers train a matched drafting head directly into models like Qwen 3.6 and Gemma 4, so llama.cpp and Ollama can enable speculation out of the box without users hunting for a compatible draft model. The core tradeoff is a simple ratio: high acceptance rates (roughly 70-80%) can roughly double throughput, while low ones barely break even or add overhead. Draft models generally need to share the main model's vocabulary and be about a tenth its size.

The technique isn't universally beneficial. Fast Mixture-of-Experts models can actually get slower because verifying draft tokens loads more expert weights than needed. Creative writing tasks see acceptance rates collapse due to unpredictable continuations. And a draft model's VRAM footprint can force the main model into a lower quantization or smaller context, erasing any gains. For engineers, the practical takeaway is to check acceptance rate and memory budget before flipping this on, since a poor pairing can make things slower, not faster.