Cold start
The delay when a serverless or scaled-to-zero endpoint has to load a model into GPU memory before it can answer the first request. Can be seconds to minutes for large models.
What it is
A warm instance already has weights resident in VRAM and replies immediately. A cold one must be scheduled onto a GPU and read tens of gigabytes of weights from storage first. Keeping an instance warm costs money even when idle - the core trade-off of serverless inference.
Why it matters
For spiky, low-volume workloads, scale-to-zero saves the hourly cost but adds a cold start to the first request after a lull. For steady traffic, a dedicated always-warm instance is usually both faster and cheaper.