VRAM
A GPU's dedicated memory. It has to hold the model weights, the KV cache for every in-flight request, and activations. When it runs out, you need a bigger GPU, more GPUs, or a smaller model.
What it is
A rough weights budget: ~2 bytes per parameter at 16-bit precision. A 70B model is ~140 GB in fp16, ~70 GB at int8, ~35 GB at int4 - see quantization. On top sits the kv cache, which grows with context and concurrency.
Why it matters
VRAM, not compute, is usually the first wall you hit when self-hosting. It decides which single GPU can run a model at all, and how many users it can serve before you need multi gpu parallelism.
Cost & infrastructure impact
Every VRAM tier up roughly doubles the hourly price (24 GB -> 48 GB -> 80 GB -> 141 GB). quantization is the main way to fit a model into a cheaper tier, at some quality cost.
Example
| Model, precision | Weights (approx) |
| 8B, fp16 | ~16 GB |
| 70B, fp16 | ~140 GB |
| 70B, int4 | ~35 GB |
Related concepts
Use it in Obolith
FAQ
How much VRAM do I need to run a 7B model?
About 16 GB in fp16 for weights plus headroom for KV cache - a 24 GB card is comfortable. At 4-bit you can fit it in ~6-8 GB.