Throughput
How much work a system does per unit time - requests per second, or total tokens per second across all concurrent users. The number that decides cost efficiency when self-hosting.
What it is
Per-user tokens per second can look modest while aggregate throughput is high, because modern inference servers process many requests in the same batch. A GPU serving 30 users at 25 tok/s each is doing 750 tok/s of useful work.
Why it matters
Cost per token on a fixed-price GPU is GPU $/hr / (aggregate tokens per hour). Low throughput - few users, small batches, an idle GPU - is what makes self hosting lose to an API.