» Tag

quantization

24 posts

Jul 27, 20269.21
397B MoE Model Runs on a Single RTX PRO 6000 96GB GPU
Krasis runtime runs a 397B-parameter MoE model on one RTX PRO 6000 96GB GPU, hitting 2,354 tok/s prefill via CPU-VRAM expert streaming.
gpu llm-inference local-llm moe
Jul 31, 20269.02
Why reformatting prompts beats quantization for CPU LLM inference
Measured on a free ARM CPU box: rewriting prompts as label:value fact sheets cut LLM prefill time 6.5x while quantization tricks hurt accuracy.
cpu inference llama.cpp llm
Jul 30, 20268.60
56M-parameter LLM runs distributed across three ESP32-S3 boards via ESP-NOW
A 56M-parameter LLM was split across three ESP32-S3 boards over ESP-NOW, using Split-PLE quantization and a KV cache for coherent generation.
edge-ai embedded-ai esp32 llm-inference
Ad
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.com
Jul 29, 20268.50
Running Kimi K3, a 2.8T-Parameter MoE Model, on an M1 Mac
Deltafin runs Kimi K3, a 2.8T-parameter MoE model, on a 64GB M1 Mac via full local install or expert streaming, no cluster required.
apple-silicon kimi-k3 llm local-inference
Jul 28, 20268.50
DeepSeek V4 Flash reaches 32 tok/s on AMD Ryzen AI MAX+ 395
AMD Ryzen AI MAX+ 395 runs 284B-parameter DeepSeek V4 Flash locally at 32 tok/s decode and ~250 tok/s sparse prefill using 128GB unified memory.
amd deepseek llm-inference local-inference
Jul 12, 20268.43
Tesla P100's silent FP16 precision bug in llama.cpp fixed
A 3-line CUDA patch in llama.cpp fixes a years-old silent FP16 precision bug on Tesla P100 GPUs, with zero performance cost.
cuda gpu-inference llama.cpp local-llm
Aug 15, 20268.41
Shoehorn: Rust Tool Quantizes LLMs to Fit Your Mac's Exact VRAM
Shoehorn is a Rust CLI that quantizes BF16 GGUF LLMs per-tensor to exactly fit your Mac's available VRAM, maximizing quality under a hard memory budget.
gguf llama-cpp llm macos
Jul 28, 20268.41
Moonshot's 2.8T-Parameter Kimi K3 Runs on a GPU-less Mini PC
A 2.8T-parameter open-weight Kimi K3 model ran on a GPU-less mini PC using disk-streamed MoE experts and native MXFP4 quantization.
edge-computing inference llm mixture-of-experts
Jul 20, 20268.41
PXQ format revives Pascal/Volta GPUs: +88% prefill on a $150 Tesla P100
PXQ format and custom CUDA kernels in ik_llama.cpp fork pxq_llama deliver major prefill/decode speedups on Tesla P100, V100 and GTX 1080 Ti GPUs.
cuda gpu-inference llm moe
Jul 28, 20268.30
Nunchaku Lite Brings 4-bit Diffusion Inference Natively to Diffusers
Nunchaku's 4-bit SVDQuant inference is now natively loadable in Diffusers via from_pretrained, no separate engine or local CUDA compile needed.
cuda diffusers diffusion-models inference-optimization