GPU
A processor with thousands of parallel cores and very fast on-package memory. The workhorse of both training and inference for large models.
What it is
For LLM work, two GPU numbers matter most: how much VRAM it has (does the model fit?) and its memory bandwidth (how fast can it stream weights, which sets tokens per second). Raw compute (TFLOPS) matters more for training and prefill than for generation.
Data-centre parts (H100, H200, B200, MI300X) have large VRAM and HBM; consumer parts (RTX 4090, 5090) are cheaper per hour but have less VRAM and no fast interconnect.
Why it matters
The GPU is the single biggest line in a self hosting budget. Picking one too small forces multi gpu or quantization; too big wastes money at low utilisation.
Cost & infrastructure impact
Rental prices for the same GPU vary 2-3x between providers and between on-demand and spot. The per-GPU-hour price is the number to compare, not the instance price.