» Tag

inference

33 posts

Jul 30, 20268.00
Scaling Laws Explained: From Kaplan to Chinchilla to Overtraining
A breakdown of LLM scaling laws from Kaplan to Chinchilla, and why modern models are deliberately overtrained to cut inference costs.
chinchilla deep-learning inference llm
Jul 25, 20268.00
Speculative Decoding: The Free Speedup Most Local LLM Setups Skip
Speculative decoding speeds up local LLM inference 1.5-2.5x with identical output; 2026 saw it built into models via multi-token prediction.
gpu inference llama.cpp llm
Jul 21, 20268.01
LLMrPro: Open-Source LLM Balancer Pools Local Machines With Cloud Fallback
LLMrPro is an open-source, self-hosted LLM balancer that unifies your own machines and cloud providers behind one OpenAI-compatible API.
cloud-fallback inference llm load-balancing
Ad
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.com
Jul 20, 20268.00
How Netflix Built Its Own In-House LLM Serving Stack
Netflix details its in-house LLM serving architecture: vLLM engine choice, Triton packaging, OpenAI-compatible API, and production rollout issues.
inference llm ml-infrastructure netflix
Jul 12, 20268.00
What Bun's Rust Migration Reveals About C# in AI Infrastructure
Bun's Zig-to-Rust rewrite shows why compiled languages matter for production AI infrastructure. TensorSharp benchmarks suggest C# can compete at the inference-engine level.
ai csharp dotnet inference
Jul 8, 20268.00
ZML/LLMD Alpha: One LLM Server Across CUDA, ROCm, TPU, Metal
ZML/LLMD alpha runs LLaMa, Gemma, Qwen and Mistral models across NVIDIA, AMD, TPU, Intel and Apple Metal in one server, with DFlash speeding up inference.
cross-platform gpu inference llm
Jul 10, 20267.81
Unified Memory: Why Mini PCs Run 70B Models a Big GPU Can't
Unified-memory mini PCs like AMD's Strix Halo can load 70B-parameter models that a $2,000 RTX 5090 cannot fit. Here's why capacity and bandwidth pull in opposite directions.
gpu hardware inference llm
Jul 7, 20267.80
OpenAI and Broadcom Unveil Jalapeño, an LLM Inference Chip
OpenAI and Broadcom unveiled Jalapeño, a custom LLM inference chip built in nine months, set for gigawatt-scale data center deployment starting in 2026.
ai broadcom hardware inference
Jul 30, 20267.60
WISP: A CUDA Engine for Streaming 744B+ Parameter MoE Models on Consumer Hardware
WISP is a CUDA engine for streaming 744B+ parameter MoE models on consumer hardware.
cuda inference machinelearning models
Jul 28, 20267.61
Ninfer: High-performance single-GPU inference
NInfer offers a high-performance C++/CUDA inference engine for RTX 5090.
cuda engineering gpu inference