» Tag
llm-inference
19 postsGLM 5.2 Local Deployment Guide: Quantization and Hardware Choices
A practical breakdown of running the 753B-parameter MoE model GLM 5.2 locally, comparing Unsloth dynamic quantizations, RAM needs, and hardware options.
Prefill/Decode Disaggregation Can Worsen Tail Latency, Not Fix It
Splitting prefill and decode across GPU pools adds queues and KV transfer overhead that can worsen tail latency without careful control-loop design.
Text Diffusion vs Autoregressive Models: The Real GPU Cost Math
A grounded comparison of text diffusion and autoregressive LLMs on GPU utilization, latency, batching economics, and real-world enterprise costs.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comRandomized KV-Cache Eviction Enables Verifiable Error Certificates
A new paper shows randomized KV-cache eviction, unlike deterministic top-k, enables verifiable per-step error certificates for LLM serving.
Bw24: From-Scratch Rust+CUDA LLM Engine Tuned for RTX 50 Blackwell
Bw24 is a from-scratch Rust+CUDA LLM inference engine tuned for RTX 50-series Blackwell GPUs, hitting up to 2.3x speculative decoding gains over llama.cpp.
Porting Gemma-4 (2B/4B/12B) to AWS Inferentia2: A Field Report
How Gemma-4 was ported to AWS Inferentia2 by bypassing vLLM/NxD, solving mixed attention heads and KV-sharing via direct model tracing.
LLM Inference Latency: Why Memory Bandwidth Beats FLOPS
Why LLM inference speed depends on GPU memory bandwidth, not FLOPS. Real tok/s benchmarks across GPUs, quantization levels, and batch sizes.
Flint: Efficiently Leveraging High Bandwidth Flash for LLM Inference
FLINT is a workload-driven hardware substrate that integrates High Bandwidth Flash as a scalable memory tier for efficient, capacity-constrained LLM inference.
Trendyol's agent tunes LLM serving configs for 4x speedup
Trendyol Tech built autooptimizer, an AI agent that autonomously tunes vLLM serving configs, achieving a 4x throughput-latency score gain on Gemma 4 26B with zero manual tweaking.