Continuous batching
An inference server technique that adds and removes requests from the running batch token-by-token, instead of waiting for a whole batch to finish. Raises GPU throughput several-fold at the same latency.
What it is
Naive ("static") batching groups N requests, runs them together until the slowest one finishes, then starts the next group - so short requests wait for long ones and the GPU idles. Continuous batching (vLLM, TGI, TensorRT-LLM) schedules at every decoding step: as soon as one request emits its final token its slot is freed and a queued request takes its place.
It depends on a well-managed kv cache to hold many half-finished requests at once.
Why it matters
Continuous batching is the single biggest reason a self-hosted GPU can serve 10-40 concurrent users instead of 2-3. It is the difference between a self-host that breaks even and one that never does. Every serious inference server does it by default now.
Cost & infrastructure impact
It does not lower your gpu hour rate - it raises the utilisation rate you can realistically hit, which is what the self-host calculators use to work out cost per request.