» Tag
vllm
14 postsTransformers vLLM Backend Now Matches Native Inference Speed
The transformers vLLM backend now matches native inference speed via torch.fx graph analysis and automatic kernel fusion for LLMs.
Single MI300X, Real Coding Agents: DeepSeek V4 Flash Throughput Tested
Real-world benchmark of DeepSeek V4 Flash on a single MI300X GPU serving coding agents, covering throughput, caching, and cost per token.
How We Self-Host DeepSeek V4 Flash on AWS Spot Instances
Inside a self-hosted DeepSeek V4 Flash deployment on AWS spot GPUs: MXFP4 quantization, RTX PRO 6000 hardware, DSpark decoding, and KV cache tuning.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comNetflix GenRec: An LLM-Native Ranker Beyond Feature Engineering
Netflix's GenRec replaces feature engineering with an LLM-native ranker, matching production performance using far less labeled data.
How Netflix Runs Its Own LLM Serving Stack with vLLM and Triton
Netflix engineers explain how they built an in-house LLM serving stack using vLLM, Triton, and an OpenAI-compatible API, with real production lessons.
Moonshot AI Ships Kimi K3: A 2.8T-Parameter Open MoE Model
Moonshot AI's Kimi K3 is a 2.8T-parameter open MoE model with 1M context and new attention and routing architectures for long-context inference.
LLM Inference Latency: Why Memory Bandwidth Beats FLOPS
Why LLM inference speed depends on GPU memory bandwidth, not FLOPS. Real tok/s benchmarks across GPUs, quantization levels, and batch sizes.
How Netflix Built Its Own In-House LLM Serving Stack
Netflix details its in-house LLM serving architecture: vLLM engine choice, Triton packaging, OpenAI-compatible API, and production rollout issues.
DiffusionGemma: Google's Diffusion-Based LLM Developer Guide
Google's DiffusionGemma developer guide: diffusion-based LLM with bidirectional attention, 4x faster generation, vLLM support, and Sudoku fine-tuning results.
Inside vLLM: Anatomy of a High-Throughput LLM Inference System
Explore the core components and features of vLLM's high-throughput LLM inference system.