Serverless inference
A hosted GPU endpoint that scales with your traffic, including down to zero. You pay per request or per second of compute, not for an always-on machine - at the cost of cold starts.
What it is
The platform loads your model on a GPU when a request arrives, keeps it warm briefly, and releases it during idle periods. Billing is usage-based (GPU-seconds or per request). Providers include Modal, Baseten, RunPod Serverless, Replicate, Together.
Why it matters
It bridges the gap between a per-token API and a dedicated GPU: cheaper than always-on for spiky traffic, more predictable than a raw API for custom models. The trade-off is cold start latency and a higher per-second rate than plain on-demand.