KV cache
Memory that stores the attention keys and values for every token already processed, so the model does not recompute them for each new output token. It lives in VRAM and grows with context length and concurrency.
What it is
To generate token N, a model attends to tokens 1..N-1. Without a cache it would reprocess the whole sequence every step. The KV cache keeps the intermediate key/value tensors for those tokens in VRAM, so each new token only needs one forward pass over itself.
Why it matters
The KV cache is why decode is fast, but it is also the main thing competing with model weights for VRAM. Its size is roughly 2 x layers x hidden_dim x context_tokens x bytes per sequence - so it scales linearly with context length and with the number of concurrent requests.
Cost & infrastructure impact
On a self-hosted model, the KV cache sets how many users you can serve at once on a given GPU. Long contexts + high concurrency can exhaust VRAM before compute is the bottleneck, forcing a bigger (more expensive) GPU or multiple GPUs.
Techniques like KV-cache quantization, paged attention and grouped-query attention exist precisely to shrink this footprint.
Example
For a 70B model, KV cache can run ~0.3-0.5 GB per 1K tokens of context per request. Serving 20 concurrent users at 16K context each is ~100-160 GB - more than a single 80 GB H100 holds alongside the weights.
Related concepts
Use it in Obolith
FAQ
Is the KV cache the same as prompt caching?
No. The KV cache is transient GPU memory used within one request. Prompt caching is a billing feature that reuses a computed prefix across separate API calls.