Speculative decoding
A speed trick: a small fast model drafts several tokens, the large model checks them all in one pass and keeps the ones it agrees with. 2-3x faster generation with identical output.
What it is
Generating one token normally means one full forward pass of the big model. Speculative decoding runs a cheap "draft" (a small model, or n-gram lookup, or extra heads like Medusa/EAGLE) to guess the next 4-8 tokens, then does one big-model pass that verifies the whole guess at once. Accepted tokens are free speed; rejected ones fall back to normal decoding. Output is provably identical to plain decoding.
Why it matters
Because it cuts wall-clock time per request without changing quality, it lifts the same GPU's throughput - more requests per gpu hour, so a lower cost per request on a self-host. Several API providers use it under the hood, which is part of why per-token prices keep falling.