Prompt caching: the first lever to pull on a RAG bill

Most teams reach for a cheaper model first. For rag and agent workloads that is the wrong first move. The bigger, easier win is prompt caching: stop paying to recompute the part of your prompt that never changes.

Why it works

When request after request starts with the same block - a system prompt, a style guide, a retrieved document, a set of tool definitions - the model does the exact same work on that block every time. If the provider stores the computed state of that prefix, a later request that matches it skips straight to the new part.

You pay a small one-time "cache write" fee, then a cache read rate of roughly 10-25% of the normal input price on every call that hits the cached prefix, within a short time window (usually 5 minutes to an hour).

How much it saves

For a typical RAG call, the repeated prefix - system prompt plus retrieved context - is 80-95% of the input tokens. The answer itself is small. So caching that prefix takes most of the input cost to near-zero.

For agents it is even larger, because the growing conversation history is re-sent every step and most of it is unchanged from the last one.

How to turn it on

• Anthropic - add cache_control breakpoints in the request; up to 4 cached segments.
• OpenAI - automatic for prompts over 1,024 tokens, no code change; the discount applies to the matched prefix.
• Google (Gemini) - explicit context caching via the API, or implicit caching on newer models.
• Open-source servers (vLLM, TGI, SGLang) - prefix caching is on by default when you self-host.

The one rule: put the stable content first (system prompt, documents, tool schemas) and the variable content last (the user turn). A cache only matches from the start of the prompt, so a changing token near the top invalidates everything after it.

When it does not help

Short prompts (a plain chatbot with a one-line system prompt), or workloads where every request is genuinely unique with no shared prefix. And a model with no cache pricing at all - Obolith's pricing table shows the cache-read rate where the provider offers one.