» Tag
llm-inference
17 postsPrefill/Decode Disaggregation Can Worsen Tail Latency, Not Fix It
Splitting prefill and decode across GPU pools adds queues and KV transfer overhead that can worsen tail latency without careful control-loop design.
Text Diffusion vs Autoregressive Models: The Real GPU Cost Math
A grounded comparison of text diffusion and autoregressive LLMs on GPU utilization, latency, batching economics, and real-world enterprise costs.
Randomized KV-Cache Eviction Enables Verifiable Error Certificates
A new paper shows randomized KV-cache eviction, unlike deterministic top-k, enables verifiable per-step error certificates for LLM serving.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comBw24: From-Scratch Rust+CUDA LLM Engine Tuned for RTX 50 Blackwell
Bw24 is a from-scratch Rust+CUDA LLM inference engine tuned for RTX 50-series Blackwell GPUs, hitting up to 2.3x speculative decoding gains over llama.cpp.
Porting Gemma-4 (2B/4B/12B) to AWS Inferentia2: A Field Report
How Gemma-4 was ported to AWS Inferentia2 by bypassing vLLM/NxD, solving mixed attention heads and KV-sharing via direct model tracing.
LLM Inference Latency: Why Memory Bandwidth Beats FLOPS
Why LLM inference speed depends on GPU memory bandwidth, not FLOPS. Real tok/s benchmarks across GPUs, quantization levels, and batch sizes.
Trendyol's agent tunes LLM serving configs for 4x speedup
Trendyol Tech built autooptimizer, an AI agent that autonomously tunes vLLM serving configs, achieving a 4x throughput-latency score gain on Gemma 4 26B with zero manual tweaking.