KV Cache Quantization's Effect on KLD in Qwen3.6-27B

A Reddit user ran a KL-divergence benchmark comparing how KV cache quantization affects output quality across Q8, Q6 and Q5 GGUF quantizations of bartowski's Qwen3.6-27B model. Using llama-perplexity with a 50K context window, chunk size of 32, and a 230MB corpus assembled from open-source Python repositories, the unquantized-KV Q8 model served as the reference baseline.

The results confirmed Q8 outperforms Q6 and Q5, but the gap between Q8 and Q6 was surprisingly small (about 0.01 units) compared to the much larger gap between Q6 and Q5. The most notable finding was that quantizing the V (value) tensor down to q4_0 causes a sharp quality drop regardless of the K tensor's setting — to the point that running Q6 with full-precision (q8_0,q8_0) KV cache can beat Q8 paired with aggressively quantized KV. Q5 turned out to be more tolerant of V quantization, and at the extreme (q4_0,q4_0) setting, Q8 and Q6 essentially converge.

The practical takeaway for engineers: pick the largest model quantization that fits in VRAM and pair it with (q8_0,q8_0) KV cache quantization, since the quality cost is nearly negligible while memory savings are real. The author cautions this is based on a single coding-focused test corpus, that the real-world significance of small KLD differences remains uncertain, and that personal experience should still weigh heavily alongside any benchmark.