Concurrency
How many requests a model is handling at the same time. Higher concurrency raises throughput and cost efficiency, but is capped by VRAM because each in-flight request needs its own KV cache.
What it is
Inference servers batch concurrent requests so one pass over the model weights serves many users. But every concurrent request holds a kv cache in VRAM for the length of its context, so there is a hard ceiling on how many fit.
Why it matters
Cost per token falls as concurrency rises - the GPU is amortised over more work. This is the main reason a shared API is cheaper per token than one lightly-used self-hosted GPU.