Latency
Total time from request to complete response. For an LLM it is roughly TTFT plus (output tokens / tokens-per-second), plus network overhead.
What it is
Latency is what one user experiences for one request. It is a different axis from throughput, which is how many requests the system handles per second across all users.
Why it matters
Batching more requests together raises throughput but can raise per-request latency - the classic trade-off. Serverless endpoints add cold start latency when they scale from zero.