LLM Quantization Guide: Comparing GPTQ, AWQ and GGUF
A practical breakdown of LLM quantization techniques - GPTQ, AWQ, GGUF and bitsandbytes - explaining how 4-bit compression cuts VRAM needs while preserving model quality.
Running large language models on consumer hardware usually hits a VRAM wall long before compute becomes the bottleneck. Quantization addresses this by representing model weights with fewer bits - typically 4-bit or 8-bit instead of 16/32-bit floats - shrinking model size, cutting memory requirements, and often speeding up inference since memory bandwidth, not raw compute, is usually the real constraint. Naive rounding fails because a small set of outlier weights carry disproportionate signal and errors compound across layers, which is why modern methods rely on calibration.
The piece walks through four major approaches: GPTQ, a mature layer-by-layer calibrated method using second-order information, popular for GPU inference; AWQ, which protects activation-critical weights and often beats GPTQ at the same bit-width; GGUF, the file format behind llama.cpp, Ollama, and LM Studio for CPU/hybrid inference, with naming conventions like Q4_K_M indicating the quality/size trade-off; and bitsandbytes, a runtime quantization library integrated into Hugging Face transformers that underpins QLoRA fine-tuning.
The practical takeaways are straightforward: 4-bit is generally the sweet spot, larger base models tolerate quantization better, format choice should match the deployment target (GPTQ/AWQ for GPU serving, GGUF for local inference), and real-task benchmarking matters more than perplexity alone. For engineers, this means production-quality LLMs are now deployable on modest hardware for local assistants, RAG pipelines, and prototyping without needing a well-funded GPU cluster.