» Tag
inference-optimization
5 postsLeakyLMs: Timing Attacks Expose LLM Architecture and Inference Tricks
LeakyLMs shows how timing side-channels can leak LLM architecture details and speculative decoding setups from production APIs like Gemini and Llama.
9.9x Lower TTFT on Real Android Phone via llama.cpp KV Reuse
EdgeSync-LLM reuses llama.cpp's public KV APIs to skip reprocessing shared prefixes, cutting Android TTFT 9.9x while catching a fake 8.8x speedup.
Single MI300X, Real Coding Agents: DeepSeek V4 Flash Throughput Tested
Real-world benchmark of DeepSeek V4 Flash on a single MI300X GPU serving coding agents, covering throughput, caching, and cost per token.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comNunchaku Lite Brings 4-bit Diffusion Inference Natively to Diffusers
Nunchaku's 4-bit SVDQuant inference is now natively loadable in Diffusers via from_pretrained, no separate engine or local CUDA compile needed.
DFlash in llama.cpp: 4.44x Faster Local Inference on Qwen 3.6 27B
DFlash, merged into llama.cpp, uses block-diffusion drafting to boost Qwen 3.6 27B inference speed, hitting 4.44x at 36K context with near-lossless quality.