Learn
The AI infrastructure glossary
Plain, technical definitions of the concepts behind AI infrastructure costs - and what each one means for your bill, your GPU choice and your architecture.
Start here
LLM & APIs 11
Batch APIAn asynchronous mode offered by most LLM providers: you submit a file of requests, get results within a deadline (usually 24 h), and pay roughly 50% of the normal per-token price.Blended token priceA single per-million-token figure that combines the input and output prices at an assumed ratio, so models can be ranked in one column. Obolith weights input 3x to output.Cached input pricingA discount on input tokens that repeat across calls - a stable system prompt, few-shot examples, a long document. The cached prefix is billed at roughly 10-25% of the normal input rate.Context windowThe maximum number of tokens a model can consider at once - prompt plus reply combined. Common sizes range from 8K to over 1M tokens.Cost per million tokensThe standard unit for LLM API pricing. Quoted separately for input and output, in USD per 1,000,000 tokens.EmbeddingA list of numbers (a vector) that represents the meaning of a piece of text, so that similar meanings sit close together. The building block of semantic search and RAG.Input tokensThe tokens you send to a model: system prompt, conversation history, retrieved context and the user message. Usually the cheaper half of the bill.Output tokensThe tokens a model generates in its reply. Usually the expensive half of the bill - 2 to 5 times the input price - because each one requires a full forward pass.Prompt cachingReusing the computed state of a repeated prompt prefix instead of recomputing it every call. On APIs it shows up as a discounted "cache read" rate; when self-hosting it is a server feature (prefix caching).RAG (Retrieval-Augmented Generation)Answering with an LLM by first retrieving relevant snippets from your own data and putting them in the prompt. Cheaper and more current than stuffing everything into a huge context window.TokenThe unit an LLM reads and writes. Roughly 3-4 characters of English, or about 0.75 words. Every API bill is counted in tokens.
Inference 17
Cold startThe delay when a serverless or scaled-to-zero endpoint has to load a model into GPU memory before it can answer the first request. Can be seconds to minutes for large models.ConcurrencyHow many requests a model is handling at the same time. Higher concurrency raises throughput and cost efficiency, but is capped by VRAM because each in-flight request needs its own KV cache.Context windowThe maximum number of tokens a model can consider at once - prompt plus reply combined. Common sizes range from 8K to over 1M tokens.Continuous batchingAn inference server technique that adds and removes requests from the running batch token-by-token, instead of waiting for a whole batch to finish. Raises GPU throughput several-fold at the same latency.InferenceRunning a trained model to get an answer, as opposed to training it. Splits into a prefill phase (reading the prompt) and a decode phase (writing the reply one token at a time).KV cacheMemory that stores the attention keys and values for every token already processed, so the model does not recompute them for each new output token. It lives in VRAM and grows with context length and concurrency.LatencyTotal time from request to complete response. For an LLM it is roughly TTFT plus (output tokens / tokens-per-second), plus network overhead.Mixture of experts (MoE)A model architecture where each token is routed to only a few of many "expert" sub-networks. Total parameters are huge, but the compute per token is small - which is why these models are cheap to serve.Multi-GPUSplitting one model across several GPUs when it does not fit in one. Adds cost and communication overhead, and needs a fast interconnect like NVLink to stay efficient.QuantizationStoring model weights (and sometimes activations or the KV cache) at lower numerical precision - 8-bit, 4-bit - to cut memory use and speed up generation, usually with a small quality cost.Self-hostingRunning an open-weight model on GPUs you rent or own, instead of calling a provider API. You trade a per-token bill for a per-hour bill plus engineering effort.Serverless inferenceA hosted GPU endpoint that scales with your traffic, including down to zero. You pay per request or per second of compute, not for an always-on machine - at the cost of cold starts.Speculative decodingA speed trick: a small fast model drafts several tokens, the large model checks them all in one pass and keeps the ones it agrees with. 2-3x faster generation with identical output.ThroughputHow much work a system does per unit time - requests per second, or total tokens per second across all concurrent users. The number that decides cost efficiency when self-hosting.Time to first token (TTFT)How long a user waits between sending a request and seeing the first word appear. Dominated by the prefill phase, so it grows with prompt length.Tokens per secondHow fast a model generates output once it has started. Sets how long a full answer takes and, on a self-hosted GPU, drives the cost per token.VRAMA GPU's dedicated memory. It has to hold the model weights, the KV cache for every in-flight request, and activations. When it runs out, you need a bigger GPU, more GPUs, or a smaller model.
GPU & hardware 9
FP16 / BF1616-bit floating-point formats. The default precision for serving large models: ~2 bytes per parameter. BF16 trades mantissa bits for range and is the training-era standard.FP88-bit floating-point. Halves memory versus 16-bit while keeping a floating-point range, so quality loss is usually smaller than integer INT8. Needs hardware support (Hopper, Blackwell, MI300).GPUA processor with thousands of parallel cores and very fast on-package memory. The workhorse of both training and inference for large models.GPU-hourThe unit for renting GPU compute: the cost of one GPU for one hour. Obolith normalises every offer to price per GPU-hour so multi-GPU instances can be compared fairly.HBMHigh Bandwidth Memory - the stacked memory used as VRAM on data-centre GPUs. Its bandwidth (TB/s), not its capacity, is what sets LLM generation speed.KV cacheMemory that stores the attention keys and values for every token already processed, so the model does not recompute them for each new output token. It lives in VRAM and grows with context length and concurrency.Multi-GPUSplitting one model across several GPUs when it does not fit in one. Adds cost and communication overhead, and needs a fast interconnect like NVLink to stay efficient.QuantizationStoring model weights (and sometimes activations or the KV cache) at lower numerical precision - 8-bit, 4-bit - to cut memory use and speed up generation, usually with a small quality cost.VRAMA GPU's dedicated memory. It has to hold the model weights, the KV cache for every in-flight request, and activations. When it runs out, you need a bigger GPU, more GPUs, or a smaller model.
Performance 10
Cold startThe delay when a serverless or scaled-to-zero endpoint has to load a model into GPU memory before it can answer the first request. Can be seconds to minutes for large models.ConcurrencyHow many requests a model is handling at the same time. Higher concurrency raises throughput and cost efficiency, but is capped by VRAM because each in-flight request needs its own KV cache.Context windowThe maximum number of tokens a model can consider at once - prompt plus reply combined. Common sizes range from 8K to over 1M tokens.InferenceRunning a trained model to get an answer, as opposed to training it. Splits into a prefill phase (reading the prompt) and a decode phase (writing the reply one token at a time).KV cacheMemory that stores the attention keys and values for every token already processed, so the model does not recompute them for each new output token. It lives in VRAM and grows with context length and concurrency.LatencyTotal time from request to complete response. For an LLM it is roughly TTFT plus (output tokens / tokens-per-second), plus network overhead.Output tokensThe tokens a model generates in its reply. Usually the expensive half of the bill - 2 to 5 times the input price - because each one requires a full forward pass.ThroughputHow much work a system does per unit time - requests per second, or total tokens per second across all concurrent users. The number that decides cost efficiency when self-hosting.Time to first token (TTFT)How long a user waits between sending a request and seeing the first word appear. Dominated by the prefill phase, so it grows with prompt length.Tokens per secondHow fast a model generates output once it has started. Sets how long a full answer takes and, on a self-hosted GPU, drives the cost per token.
Optimisation 11
Batch APIAn asynchronous mode offered by most LLM providers: you submit a file of requests, get results within a deadline (usually 24 h), and pay roughly 50% of the normal per-token price.Cached input pricingA discount on input tokens that repeat across calls - a stable system prompt, few-shot examples, a long document. The cached prefix is billed at roughly 10-25% of the normal input rate.Continuous batchingAn inference server technique that adds and removes requests from the running batch token-by-token, instead of waiting for a whole batch to finish. Raises GPU throughput several-fold at the same latency.FP16 / BF1616-bit floating-point formats. The default precision for serving large models: ~2 bytes per parameter. BF16 trades mantissa bits for range and is the training-era standard.FP88-bit floating-point. Halves memory versus 16-bit while keeping a floating-point range, so quality loss is usually smaller than integer INT8. Needs hardware support (Hopper, Blackwell, MI300).INT44-bit integer quantization. Cuts model size to about a quarter of 16-bit. Big cost win, but quality loss is more noticeable and model-dependent - test before relying on it.INT88-bit integer quantization. Roughly halves model size versus 16-bit with small, often negligible quality loss for chat-style workloads. Widely supported.Mixture of experts (MoE)A model architecture where each token is routed to only a few of many "expert" sub-networks. Total parameters are huge, but the compute per token is small - which is why these models are cheap to serve.Prompt cachingReusing the computed state of a repeated prompt prefix instead of recomputing it every call. On APIs it shows up as a discounted "cache read" rate; when self-hosting it is a server feature (prefix caching).QuantizationStoring model weights (and sometimes activations or the KV cache) at lower numerical precision - 8-bit, 4-bit - to cut memory use and speed up generation, usually with a small quality cost.Speculative decodingA speed trick: a small fast model drafts several tokens, the large model checks them all in one pass and keeps the ones it agrees with. 2-3x faster generation with identical output.
Cloud infrastructure 9
API vs self-hostingThe core infrastructure decision for anyone running LLMs at scale: pay per token to a provider, or pay per hour for GPUs you operate. The answer turns on volume, utilisation and how much control you need.Cold startThe delay when a serverless or scaled-to-zero endpoint has to load a model into GPU memory before it can answer the first request. Can be seconds to minutes for large models.EgressThe fee cloud providers charge to move data out of their network to the internet. Storage is cheap; getting your data back out is where object-storage bills add up.GPU-hourThe unit for renting GPU compute: the cost of one GPU for one hour. Obolith normalises every offer to price per GPU-hour so multi-GPU instances can be compared fairly.On-demand instanceA cloud GPU you rent by the hour or second with no commitment, at the standard rate. Available immediately, billed only while running, but the most expensive per hour.Self-hostingRunning an open-weight model on GPUs you rent or own, instead of calling a provider API. You trade a per-token bill for a per-hour bill plus engineering effort.Serverless inferenceA hosted GPU endpoint that scales with your traffic, including down to zero. You pay per request or per second of compute, not for an always-on machine - at the cost of cold starts.Spot instanceSpare cloud GPU capacity sold at a steep discount (often 50-80% off on-demand) that the provider can reclaim with little notice. Good for interruptible work, risky for live serving.Utilisation rateThe share of time a rented GPU is doing useful work rather than sitting idle. The single number that decides whether self-hosting beats an API.
Training & fine-tuning 7
EpochOne full pass of the training set through the model. Fine-tuning runs are usually 1-4 epochs; each extra epoch multiplies the GPU cost of the run and, past a point, causes overfitting.Fine-tuningContinuing to train an existing model on your own examples so it adapts to a task, style or format. Cheaper and faster than training from scratch; still a GPU cost you pay once per run.Full fine-tuningFine-tuning that updates every weight in the model, not a small adapter. Highest quality ceiling, but needs several times the model size in VRAM and usually a multi-GPU node.LoRALow-Rank Adaptation: fine-tuning that freezes the base model and trains a small pair of matrices per layer instead. ~0.1-1% as many trainable parameters, so it fits on one GPU and runs in minutes to hours.Multi-GPUSplitting one model across several GPUs when it does not fit in one. Adds cost and communication overhead, and needs a fast interconnect like NVLink to stay efficient.Preference tuning (RLHF, DPO)A second training stage that teaches a model which of two answers people prefer, rather than a single target answer. RLHF uses a reward model + reinforcement learning; DPO does it directly and cheaply.QLoRALoRA on top of a base model that has been quantised to 4-bit. Cuts the VRAM needed to fine-tune by ~3-4x, so a 70B model tunes on a single 48 GB GPU instead of a multi-GPU node.
Embeddings & vector DBs 5
ChunkingSplitting documents into smaller passages before embedding them for RAG. Chunk size trades retrieval precision against context completeness, and directly affects vector count and cost.EmbeddingA list of numbers (a vector) that represents the meaning of a piece of text, so that similar meanings sit close together. The building block of semantic search and RAG.Embedding modelA model specialised in producing embeddings rather than generating text. Chosen on retrieval quality (MTEB score), vector dimensions, max input length and price.RAG (Retrieval-Augmented Generation)Answering with an LLM by first retrieving relevant snippets from your own data and putting them in the prompt. Cheaper and more current than stuffing everything into a huge context window.Vector databaseA database built to store embeddings and find the nearest ones to a query vector fast. The recurring cost centre of a RAG system.
Storage 2
EgressThe fee cloud providers charge to move data out of their network to the internet. Storage is cheap; getting your data back out is where object-storage bills add up.Vector databaseA database built to store embeddings and find the nearest ones to a query vector fast. The recurring cost centre of a RAG system.
Economics & pricing 22
API vs self-hostingThe core infrastructure decision for anyone running LLMs at scale: pay per token to a provider, or pay per hour for GPUs you operate. The answer turns on volume, utilisation and how much control you need.Batch APIAn asynchronous mode offered by most LLM providers: you submit a file of requests, get results within a deadline (usually 24 h), and pay roughly 50% of the normal per-token price.Blended token priceA single per-million-token figure that combines the input and output prices at an assumed ratio, so models can be ranked in one column. Obolith weights input 3x to output.Break-even pointThe volume or utilisation level at which two options cost the same - typically API vs self-hosting, or renting vs buying a GPU. Below it, one option wins; above it, the other.Cached input pricingA discount on input tokens that repeat across calls - a stable system prompt, few-shot examples, a long document. The cached prefix is billed at roughly 10-25% of the normal input rate.ConcurrencyHow many requests a model is handling at the same time. Higher concurrency raises throughput and cost efficiency, but is capped by VRAM because each in-flight request needs its own KV cache.Cost per million tokensThe standard unit for LLM API pricing. Quoted separately for input and output, in USD per 1,000,000 tokens.EgressThe fee cloud providers charge to move data out of their network to the internet. Storage is cheap; getting your data back out is where object-storage bills add up.GPU-hourThe unit for renting GPU compute: the cost of one GPU for one hour. Obolith normalises every offer to price per GPU-hour so multi-GPU instances can be compared fairly.Input tokensThe tokens you send to a model: system prompt, conversation history, retrieved context and the user message. Usually the cheaper half of the bill.Mixture of experts (MoE)A model architecture where each token is routed to only a few of many "expert" sub-networks. Total parameters are huge, but the compute per token is small - which is why these models are cheap to serve.On-demand instanceA cloud GPU you rent by the hour or second with no commitment, at the standard rate. Available immediately, billed only while running, but the most expensive per hour.Output tokensThe tokens a model generates in its reply. Usually the expensive half of the bill - 2 to 5 times the input price - because each one requires a full forward pass.Prompt cachingReusing the computed state of a repeated prompt prefix instead of recomputing it every call. On APIs it shows up as a discounted "cache read" rate; when self-hosting it is a server feature (prefix caching).RAG (Retrieval-Augmented Generation)Answering with an LLM by first retrieving relevant snippets from your own data and putting them in the prompt. Cheaper and more current than stuffing everything into a huge context window.Self-hostingRunning an open-weight model on GPUs you rent or own, instead of calling a provider API. You trade a per-token bill for a per-hour bill plus engineering effort.Serverless inferenceA hosted GPU endpoint that scales with your traffic, including down to zero. You pay per request or per second of compute, not for an always-on machine - at the cost of cold starts.Spot instanceSpare cloud GPU capacity sold at a steep discount (often 50-80% off on-demand) that the provider can reclaim with little notice. Good for interruptible work, risky for live serving.ThroughputHow much work a system does per unit time - requests per second, or total tokens per second across all concurrent users. The number that decides cost efficiency when self-hosting.TokenThe unit an LLM reads and writes. Roughly 3-4 characters of English, or about 0.75 words. Every API bill is counted in tokens.Tokens per secondHow fast a model generates output once it has started. Sets how long a full answer takes and, on a self-hosted GPU, drives the cost per token.Utilisation rateThe share of time a rented GPU is doing useful work rather than sitting idle. The single number that decides whether self-hosting beats an API.