Multi-GPU
Splitting one model across several GPUs when it does not fit in one. Adds cost and communication overhead, and needs a fast interconnect like NVLink to stay efficient.
What it is
Tensor parallelism splits each layer's matrices across GPUs that then exchange partial results every step - fast, but only if the GPUs share a high-bandwidth link (NVLink / NVSwitch). Pipeline parallelism puts different layers on different GPUs; cheaper to connect, but can leave GPUs idle waiting for each other.
Why it matters
Going from 1 to 2 GPUs more than doubles cost (two GPUs + interconnect + coordination loss). Fitting the model on one GPU via quantization is almost always cheaper if quality allows.