Quantization
A technique that reduces the numerical precision of a model's parameters — trading small amounts of quality for large reductions in memory and compute.
A trained model stores its parameters as floating-point numbers, typically 16 or 32 bits each. Quantization compresses these numbers to lower precision — 8-bit, 4-bit, sometimes as low as 2-bit — so the same model file becomes much smaller and faster to run.
The tradeoff: quality degrades as precision drops. 8-bit quantization is usually indistinguishable from the full-precision original on most tasks; 4-bit shows measurable but often acceptable degradation; 2-bit and below have noticeable quality loss on hard tasks. Different quantization techniques recover different amounts of the lost quality — GPTQ, AWQ, GGUF, and others are the popular options in 2026.
Quantization is why running a 70-billion-parameter model on a laptop is possible in 2026. A full-precision Llama 70B is ~140GB and requires expensive server hardware; a 4-bit quantized version is ~40GB and runs (slowly) on a high-end consumer machine.