Inference
Running a trained model to get an answer, as opposed to training it. Splits into a prefill phase (reading the prompt) and a decode phase (writing the reply one token at a time).
What it is
Prefill processes the whole prompt in one parallel pass and builds the kv cache. Its cost scales with prompt length and it sets time to first token.
Decode then generates the answer one token at a time, each step reading the growing KV cache. It is memory-bandwidth bound and sets tokens per second.
Why it matters
Inference is what an API bill and a self-hosted GPU both pay for. Understanding the two phases explains why long prompts are slow to start, why generation speed barely depends on prompt length, and why concurrency is limited by VRAM.
Cost & infrastructure impact
On a self-hosted GPU, cost per token is entirely a function of how many tokens/hour you push through versus the hourly price - see utilisation rate and break even point. Batching many requests together is the main way to raise that throughput.