Shoehorn: Rust Tool Quantizes LLMs to Fit Your Mac's Exact VRAM

Shoehorn is an open-source Rust CLI that quantizes BF16 GGUF language models to precisely match a machine's actual available VRAM, instead of relying on generic presets like Q4_K_M or Q5_K_S. It probes Apple Silicon's Metal API for real usable GPU memory, subtracts KV cache and compute buffer requirements, then solves a per-tensor mixed-precision assignment via Lagrangian relaxation plus a greedy fill pass — reaching over 99.9% budget utilization while maximizing quality per byte, guided by an importance matrix (imatrix).

The tool re-implements ggml's K-quant and IQ quantization objectives from scratch without linking any llama.cpp code, using llama.cpp only as the inference backend and a correctness oracle. Its `fit` command runs the full pipeline (download, imatrix generation, solve, serve) in one step, `vram` probes capacity, `quantize`/`plan` offer manual control, and `run` launches llama-server. Output is standard GGUF v3, directly loadable by any llama.cpp-based tool.

For engineers, this closes the gap between generic quantization tiers and actual hardware limits, extracting maximum model quality from a fixed memory budget instead of guessing which preset will fit.