FP16 / BF16
16-bit floating-point formats. The default precision for serving large models: ~2 bytes per parameter. BF16 trades mantissa bits for range and is the training-era standard.
What it is
FP16 and BF16 both use 16 bits. FP16 has more precision but a narrow range (can overflow); BF16 has the same range as 32-bit float with less precision, which makes it more stable for training. For inference, either gives full-quality output at 2 bytes/param.
Why it matters
This is the baseline the vram budget and quantization savings are measured against. "The 70B model needs 140 GB" assumes fp16/bf16.