HBM
High Bandwidth Memory - the stacked memory used as VRAM on data-centre GPUs. Its bandwidth (TB/s), not its capacity, is what sets LLM generation speed.
What it is
Generating each output token means reading the entire model (or a quantized copy of it) plus the kv cache out of memory once. So tokens/sec ~= memory bandwidth / bytes read per token. An H200 (~4.8 TB/s) generates faster than an H100 (~3.35 TB/s) at the same model, even though compute is similar.
Why it matters
When comparing GPUs for inference, bandwidth predicts tokens per second better than TFLOPS. Consumer cards have far less bandwidth, which is a big part of why they generate slowly.