» Tag
gguf
5 postsShoehorn: Rust Tool Quantizes LLMs to Fit Your Mac's Exact VRAM
Shoehorn is a Rust CLI that quantizes BF16 GGUF LLMs per-tensor to exactly fit your Mac's available VRAM, maximizing quality under a hard memory budget.
llama.cpp Adds Lossless F32 Compression via QFX32/QFX16
llama.cpp PR adds QFX32/QFX16 GGUF formats that losslessly compress F32 models by 2.05x with bit-identical weight reconstruction.
LLM Quantization Guide: Comparing GPTQ, AWQ and GGUF
A practical breakdown of LLM quantization techniques - GPTQ, AWQ, GGUF and bitsandbytes - explaining how 4-bit compression cuts VRAM needs while preserving model quality.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comKV Cache Quantization's Effect on KLD in Qwen3.6-27B
A KL-divergence benchmark on bartowski's Qwen3.6-27B GGUF quants (Q8/Q6/Q5) shows KV cache quantization at (q8_0,q8_0) preserves quality almost for free.
Camelid: Local AI Inference in Rust with Multiple Interfaces
Camelid enables local AI inference in Rust with multiple interfaces, running GGUF models directly.