Tokens per second
How fast a model generates output once it has started. Sets how long a full answer takes and, on a self-hosted GPU, drives the cost per token.
What it is
A per-request figure for the decode phase. Each output token needs one pass that reads the model weights and the kv cache from VRAM, so speed is set by memory bandwidth, not raw compute.
It barely depends on prompt length - that cost lands in ttft instead.
Why it matters
A 300-token answer at 20 tok/s takes 15 seconds; at 120 tok/s, 2.5 seconds. For agents that chain many model calls, slow generation compounds into minutes.
Cost & infrastructure impact
On a rented GPU, cost per 1M tokens = (GPU $/hr / tok/s) x 278. Doubling tok/s - via a faster GPU, quantization, or better batching - halves the per-token cost. This is the core of the self hosting arithmetic.
Related concepts
Use it in Obolith
FAQ
Why is my self-hosted model slower than the API?
Providers run tuned inference stacks, batch aggressively across many users, and often serve quantized weights on top-tier hardware. A single unbatched request on one GPU rarely matches that.