What does it really cost to run an LLM?

There is no single "price of an LLM". There is a price per input token, a different price per output token, a discounted price for cached input, a half-price batch lane, and - if you rent the GPU yourself - a price per hour that has nothing to do with tokens at all. Your bill is whichever combination your workload lands on.

This guide walks through each layer, with the arithmetic, so you can predict a bill before you run it.

1. The unit: dollars per million tokens

A token is roughly ¾ of a word. Providers quote two numbers per model, e.g. $0.50 / $1.50 per 1M - fifty cents per million tokens you send, a dollar fifty per million it generates. Your monthly cost for the LLM call is:

Output is almost always 2-5x the price of input, because generating a token requires a full forward pass of the model while reading one is cheap. That asymmetry is the whole reason a "blended" price is misleading - more on that below.

Obolith normalises every model to this unit and adds a 3:1 blended figure so you can sort ~200 models in one column, but the calculator uses your real ratio.

2. The same model, three different bills

A chatbot, a rag app and an agent send very different shapes of request. Here is what one request costs on the same mid-tier model, and what dominates:

• Chatbot - short prompt, short reply. Output is the bigger half. Lever: a cheaper model, or cap max_tokens.
• RAG - a few thousand tokens of retrieved context per call, a short answer. Input is 80-95% of the cost. Lever: prompt caching on the parts that repeat.
• Agent - each step re-sends the whole conversation plus tool results, and generates reasoning. Both axes balloon. Lever: caching, a smaller model for routing steps, and fewer steps.

This is why the cheapest sticker price rarely wins for you. Model your real numbers - the monthly API cost calculator applies your traffic to every model at once.

3. Two discounts most teams miss

Cached input. When many requests start with the same block - a system prompt, a style guide, a retrieved document - the provider can reuse the computation. You pay a small write fee, then roughly 10-25% of the input rate on every later call that hits the prefix. For RAG and agents this is the single biggest lever; see cached input pricing.

The batch API. Submit a file of requests, get results within 24 hours, pay about half. Anything not in front of a user in real time - nightly summaries, bulk classification, embedding a corpus, evals - belongs here. See batch api.

4. Where the rest of the stack fits

The LLM call is usually the largest line, but a production RAG stack touches four more paid layers. For a reference workload they land roughly like this:

The vector database line is the one that surprises people: serverless plans bill storage + queries, dedicated plans bill compute time, and the crossover is a few million vectors. embedding cost is a one-off for the initial index plus a trickle per query. Object storage for the raw documents is cents unless you re-index constantly (egress is the swing factor). Tracing tools like Langfuse and Helicone have free tiers that cover you until real volume.

5. When a rented GPU beats the API

An API charges for tokens. A self-hosted model charges for every hour the machine is on, busy or idle. So self-hosting is a bet on utilisation - the fraction of time your GPU does useful work.

Read it left to right. Small volume: the API is far cheaper - you are not paying for idle silicon. As volume grows the API bill climbs a straight line while the GPU bill sits flat until you need a second card. Past the crossover the GPU wins, and keeps winning.

Two things move the crossover. Model openness: only open-weight models (Llama, Qwen, DeepSeek, Mistral, gpt-oss…) can be self-hosted - a frontier closed model has no GPU option. Traffic shape: steady business-hours load hits ~55% utilisation; a bursty or mostly-idle service hits 10-30%, which pushes the break-even far out. The Stack cost tool asks for your traffic profile and does this per model.

If the math says self-host wins, the next question is what to actually buy. The AI infrastructure builder turns a GPU count into a real parts list - price, power draw and which models it can run.

6. A checklist

• Quote every model in $ / 1M in and $ / 1M out separately, never a single number.
• Multiply by your token counts and request volume - use the calculator.
• Turn on prompt caching before you compare providers. It is the biggest single lever for RAG and agents.
• Move anything that can wait to the batch API.
• Add the non-LLM layers (vector DB, storage, tracing) - usually 15-25% on top.
• Only consider self-hosting if the model is open-weight and you can keep the GPU busy. Below a few million requests a month, the API almost always wins.

The cheapest viable stack is rarely the one with the lowest sticker price. It is the one that fits your workload shape.