» Tag
quantization
24 postsllama.cpp Adds Lossless F32 Compression via QFX32/QFX16
llama.cpp PR adds QFX32/QFX16 GGUF formats that losslessly compress F32 models by 2.05x with bit-identical weight reconstruction.
Quantprobe Runs a 110B-Parameter LLM on a 16GB RAM 2016 Desktop
Quantprobe shows how careful memory-tier placement lets a 2016 desktop with 16GB RAM run a 110B-parameter LLM, validated with pre-registered predictions.
MoE Expert Streaming Runs a 120B-Parameter LLM on an Android Phone
An open-source Android app streams MoE experts from flash storage to run a 120B-parameter LLM on phone CPUs, with 30B models at usable speeds.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comQwisp: MoE expert-streaming engine runs Qwen3.6-35B-A3B on 8GB Macs
Qwisp streams MoE experts from flash to run the 35B-parameter Qwen3.6-A3B model on 8GB Macs, with bit-exact lossless decoding and raw-Metal speed.
Intel Arc B70 32GB Runs Qwen3.6-35B-A3B at 130 t/s via Vulkan
Benchmark: Intel Arc Pro B70 32GB runs Qwen3.6-35B-A3B via llama.cpp Vulkan at ~130 t/s (4-bit, 262k ctx) and ~69 t/s hybrid 8-bit offload.
GLM 5.2 Local Deployment Guide: Quantization and Hardware Choices
A practical breakdown of running the 753B-parameter MoE model GLM 5.2 locally, comparing Unsloth dynamic quantizations, RAM needs, and hardware options.
Probelock: A Lockfile for LLM Tool-Calling Capabilities
Probelock measures an LLM's tool-calling capabilities with deterministic tests and halts CI when a model, quantization, or runtime swap causes a regression.
LLM Inference Latency: Why Memory Bandwidth Beats FLOPS
Why LLM inference speed depends on GPU memory bandwidth, not FLOPS. Real tok/s benchmarks across GPUs, quantization levels, and batch sizes.
LLM Quantization Guide: Comparing GPTQ, AWQ and GGUF
A practical breakdown of LLM quantization techniques - GPTQ, AWQ, GGUF and bitsandbytes - explaining how 4-bit compression cuts VRAM needs while preserving model quality.
antirez's DS4 engine: DeepSeek V4 Flash tops local coding on a MacBook Pro
DeepSeek V4 Flash via antirez's DS4 engine one-shots complex coding prompts on a MacBook Pro, outperforming other local models in real-world testing.