» Tag
kv-cache
6 posts9.9x Lower TTFT on Real Android Phone via llama.cpp KV Reuse
EdgeSync-LLM reuses llama.cpp's public KV APIs to skip reprocessing shared prefixes, cutting Android TTFT 9.9x while catching a fake 8.8x speedup.
The Real Cost of LLM Inference Is Memory Bandwidth
LLM serving costs are driven by memory bandwidth, not FLOPs. Why KV cache movement, prefill/decode disaggregation, and batching define real inference economics.
Prefill/Decode Disaggregation Can Worsen Tail Latency, Not Fix It
Splitting prefill and decode across GPU pools adds queues and KV transfer overhead that can worsen tail latency without careful control-loop design.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comRandomized KV-Cache Eviction Enables Verifiable Error Certificates
A new paper shows randomized KV-cache eviction, unlike deterministic top-k, enables verifiable per-step error certificates for LLM serving.
TurboQuant: Four Months After Google's 6x VRAM Claim
Google's TurboQuant algorithm aimed for a 6x memory reduction, but official code is still not released.
KV Cache Quantization's Effect on KLD in Qwen3.6-27B
A KL-divergence benchmark on bartowski's Qwen3.6-27B GGUF quants (Q8/Q6/Q5) shows KV cache quantization at (q8_0,q8_0) preserves quality almost for free.