Context window
The maximum number of tokens a model can consider at once - prompt plus reply combined. Common sizes range from 8K to over 1M tokens.
What it is
The context window is the model's working memory, measured in tokens. Everything the model reasons over - system prompt, chat history, retrieved documents, the current question, and the answer it is writing - must fit inside it. Go over and the request is rejected or the oldest tokens are dropped.
A stated 128K window does not mean 128K of free input: the reply competes for the same budget, and some providers reserve headroom.
Why it matters
A larger window lets you skip retrieval and just paste documents in - simpler to build, but you pay input for every token every call. It also raises latency: time to first token grows with prompt length because the model must process it all before replying.
Cost & infrastructure impact
Large context is a cost trap when the same documents are sent repeatedly. Two mitigations: prompt caching for a stable prefix, and RAG to send only the relevant few thousand tokens instead of the whole corpus.
Some models (e.g. Gemini) also charge a higher per-token rate above a context threshold.
Example
Pasting a 50-page contract (~40K tokens) into every question in a 20-turn review means ~800K input tokens for the session. Retrieving 3 relevant clauses per question (~1.5K tokens) would be ~30K - a ~25x reduction.
Related concepts
Use it in Obolith
FAQ
Does a bigger context window make a model smarter?
Not directly. It can hold more information, but models often struggle to use the middle of a very long context well ("lost in the middle"). Relevant, focused context usually beats a large dump.