Is it cheaper to self-host, or just use an API?

Renting an H100 by the hour looks cheap next to a per-token API bill - until you account for the hours that GPU sits idle. Here is the arithmetic that decides it.

An API charges you only for tokens you actually send and receive. A self-hosted model charges you for every second the machine is on, whether it is serving a request or waiting for one. So the question is really about utilisation: how much of the time your GPU does useful work.

The break-even, in one line

Take the hourly price of the GPU, divide by the tokens per hour it can produce at your batch size, and compare that to the API’s price per token.

Plug in real numbers. An H100 at $2.50/hr serving a 70B model at 2.0M output tokens per hour, running at 40% utilisation:

At 40% utilisation the API wins by a wide margin. Self-hosting only pulls ahead once you keep the GPU busy most of the time - roughly 70%+ for this model and price - or once data residency, latency, or fine-tuned weights make the API a non-option.

Idle GPU time is the whole cost story. Everything else is a rounding error.

Two more costs the formula leaves out, both in the API’s favour: the engineering time to run inference reliably, and the capacity you pay for but never use during traffic troughs. Reserved GPUs lower the hourly rate but raise the utilisation bar.