Quantization
Storing model weights (and sometimes activations or the KV cache) at lower numerical precision - 8-bit, 4-bit - to cut memory use and speed up generation, usually with a small quality cost.
What it is
Models are trained in 16-bit (fp16 / bf16). Quantization compresses the stored weights to 8-bit (~half the size) or 4-bit (~quarter). Because inference generation is memory-bound, smaller weights also mean faster tokens per second.
Modern schemes (AWQ, GPTQ, GGUF k-quants, fp8) keep quality loss small for many models, but it is model- and task-dependent - reasoning and code are more sensitive than chat.
Why it matters
Quantization is the single biggest lever for cheap self hosting: a 4-bit 70B model can run on one 48 GB GPU instead of two 80 GB ones. That can turn a losing self-host case into a winning one.
Cost & infrastructure impact
There is no universally optimal level. Lower precision reduces memory and cost but may hurt quality depending on the model and workload - test on your own eval set before committing.