» Tag
inference
44 postsWhy reformatting prompts beats quantization for CPU LLM inference
Measured on a free ARM CPU box: rewriting prompts as label:value fact sheets cut LLM prefill time 6.5x while quantization tricks hurt accuracy.
Mozilla Report: Open-Source AI Wins Tokens, Lags in Production
Mozilla's 2026 report shows open-source AI models leading in token volume but trailing closed models in production deployment.
Colibri lets 744B-parameter GLM-5.2 run on just 25GB of RAM
Colibri is a single-file C engine that runs GLM-5.2's 744B MoE model on 25GB RAM with no GPU, streaming experts from NVMe on demand.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comUAP: An Open Ad Protocol That Can't Touch LLM Answers
UAP is an open, vendor-neutral protocol letting any LLM provider sell ads without altering answers or leaking conversation context.
NVIDIA ModelExpress: P2P RDMA Cuts Model Startup From Minutes to Seconds
NVIDIA ModelExpress speeds up LLM weight loading with P2P GPU-to-GPU RDMA transfers, cutting model startup time from minutes to seconds.
Moonshot's 2.8T-Parameter Kimi K3 Runs on a GPU-less Mini PC
A 2.8T-parameter open-weight Kimi K3 model ran on a GPU-less mini PC using disk-streamed MoE experts and native MXFP4 quantization.
llama.cpp Adds Lossless F32 Compression via QFX32/QFX16
llama.cpp PR adds QFX32/QFX16 GGUF formats that losslessly compress F32 models by 2.05x with bit-identical weight reconstruction.
How Netflix Runs Its Own LLM Serving Stack with vLLM and Triton
Netflix engineers explain how they built an in-house LLM serving stack using vLLM, Triton, and an OpenAI-compatible API, with real production lessons.
LLM Quantization Guide: Comparing GPTQ, AWQ and GGUF
A practical breakdown of LLM quantization techniques - GPTQ, AWQ, GGUF and bitsandbytes - explaining how 4-bit compression cuts VRAM needs while preserving model quality.
Benchmarking Inference Energy Costs of LLM: A LLaMA Study
Researchers benchmark LLaMA model sizes on V100 and A100 GPUs to analyze the energy and compute costs of LLM inference at scale.