Utilisation rate
The share of time a rented GPU is doing useful work rather than sitting idle. The single number that decides whether self-hosting beats an API.
What it is
An API bills only for tokens. A rented GPU bills for every hour it is on, working or not. If your traffic keeps a GPU busy 30% of the day, you pay for 100% and waste 70%. Effective cost per token is sticker cost / utilisation.
Why it matters
Bursty or small workloads rarely clear the ~60-70% utilisation needed to beat a modern API. Steady, high-volume, predictable traffic is where self-hosting wins. autoscaling and spot pricing help; a dedicated always-on GPU for a spiky app is the classic mistake.