» Tag
llm-inference
19 postsPorting Gemma-4 (2B/4B/12B) to AWS Inferentia2: A Field Report
How Gemma-4's mixed attention heads and KV-sharing broke AWS's vendor stack, and the tracing/sharding fix that made Inferentia2 inference work.
397B MoE Model Runs on a Single RTX PRO 6000 96GB GPU
Krasis runtime runs a 397B-parameter MoE model on one RTX PRO 6000 96GB GPU, hitting 2,354 tok/s prefill via CPU-VRAM expert streaming.
Transformers vLLM Backend Now Matches Native Inference Speed
The transformers vLLM backend now matches native inference speed via torch.fx graph analysis and automatic kernel fusion for LLMs.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.com56M-parameter LLM runs distributed across three ESP32-S3 boards via ESP-NOW
A 56M-parameter LLM was split across three ESP32-S3 boards over ESP-NOW, using Split-PLE quantization and a KV cache for coherent generation.
SwarmLLM: Peer-to-Peer LLM Inference Across Browser Tabs
SwarmLLM splits Qwen 3.8 27B across browser tabs via WebRTC and a custom WebGPU engine, matching native llama.cpp decode speed with zero install.
DeepSeek V4 Flash reaches 32 tok/s on AMD Ryzen AI MAX+ 395
AMD Ryzen AI MAX+ 395 runs 284B-parameter DeepSeek V4 Flash locally at 32 tok/s decode and ~250 tok/s sparse prefill using 128GB unified memory.
How We Self-Host DeepSeek V4 Flash on AWS Spot Instances
Inside a self-hosted DeepSeek V4 Flash deployment on AWS spot GPUs: MXFP4 quantization, RTX PRO 6000 hardware, DSpark decoding, and KV cache tuning.
The Real Cost of LLM Inference Is Memory Bandwidth
LLM serving costs are driven by memory bandwidth, not FLOPs. Why KV cache movement, prefill/decode disaggregation, and batching define real inference economics.
Apple Silicon macOS VMs Get 11-16x Faster LLM Inference via Llama.cpp
Cua's Metal capability shim delivers 11-16x faster llama.cpp LLM inference in macOS VMs on Apple Silicon, with open benchmarks and source.
Headroom: browser tool measures your GPU's true bandwidth ceiling for local AI
Headroom is a no-account WebGPU browser tool measuring real GPU bandwidth ceilings and driver bugs for local AI inference performance.