Most teams reason about LLMs as a FLOPs problem. That intuition is correct for training and almost entirely wrong for serving. When you serve a decoder-only transformer at scale, the binding constraints are HBM capacity and HBM bandwidth, not tensor-core throughput. A modern GPU can do hundreds of TFLOPs, but during autoregressive decode it spends most of its time waiting on memory. Understanding why that is true, and engineering around it, is the difference between $0.20 and $2.00 per million tokens.
This post walks the full serving stack the way we build it: the KV-cache memory problem, paged attention and block allocation, continuous batching, the prefill/decode split, speculative decoding, quantization trade-offs, and tensor parallelism. Throughout, the lens is the same: what decides cost-per-token, and what decides tail latency. We serve production traffic on vLLM, NVIDIA Triton, and ONNX Runtime with our own GPU orchestration layer, so the numbers here are the ones we actually optimize against.
The KV-cache is the whole ballgame
In a decoder transformer, attention at step t needs the keys and values of every previous token. Recomputing them each step would be quadratic, so we cache them. That KV-cache is the dominant consumer of VRAM once you are past the model weights, and it grows linearly with everything you care about: batch size, sequence length, layer count, and KV-head count.
The per-token, per-request cache size is 2 (K and V) x num_layers x num_kv_heads x head_dim x bytes_per_element. Multiply by sequence length and batch to get the live footprint. For a 70B-class model with 80 layers, 8 KV heads (grouped-query attention), and head_dim 128 in FP16, one token costs 2 x 80 x 8 x 128 x 2 bytes = 327,680 bytes, roughly 320 KiB per token. That is the number that quietly eats your GPU.
Weights are a fixed cost you pay once. The KV-cache is a variable cost you pay for every token of every concurrent request, and it is what actually determines how many users fit on a GPU.
Now stretch the sequence. At 4K context that request holds about 1.25 GiB of KV. At 256K context it is 320 KiB x 256K = ~78 GiB for a single sequence, more than an entire 80 GB H100 can hold, before you have loaded a single weight. This is the long-context KV explosion, and it is why naive long-context serving is economically impossible. GQA already cut KV heads from 64 to 8 (an 8x reduction versus multi-head); without it, that same request would need over 600 GiB.
Paged attention: virtual memory for the cache
Classic serving pre-allocated a contiguous KV buffer sized to max sequence length for every slot in the batch. If a request could theoretically reach 8K tokens, you reserved 8K worth of KV even if it emitted 200. Measured internal and external fragmentation routinely wasted 60 to 80 percent of KV memory. That waste directly lowers the batch size you can run, which directly lowers throughput.
Paged attention, introduced by vLLM, applies operating-system virtual memory to the cache. KV is stored in fixed-size blocks (commonly 16 tokens each). A per-request block table maps logical token positions to physical blocks scattered anywhere in HBM. Blocks are allocated on demand as the sequence grows, so you only pay for tokens that actually exist. Fragmentation drops to at most one partial block per sequence.
- Near-zero waste: memory is committed per generated token, not per worst-case length, often 2-4x more concurrent requests on the same card.
- Copy-on-write sharing: a shared system prompt or a beam-search fork points multiple block tables at the same physical blocks until one diverges.
- Prefix caching: identical prompt prefixes across requests reuse cached blocks, eliminating redundant prefill for RAG and few-shot templates.
- The cost is a custom attention kernel that gathers non-contiguous blocks, but the memory win dwarfs the indirection overhead.
Prefill and decode are two different machines
An inference request has two phases with opposite hardware profiles. Prefill processes the entire prompt in one forward pass: every token is available, so the matmuls are large and dense. Prefill is compute-bound and saturates the tensor cores. Decode generates one token at a time; each step multiplies a single new token's activations against the full weight matrices and reads the entire KV-cache back from HBM.
That asymmetry is the core of serving economics. In decode, arithmetic intensity is tiny: you move gigabytes of weights and KV per step to compute a single token, so you are memory-bandwidth-bound. A step's floor latency is roughly (model bytes read + KV bytes read) / HBM bandwidth. On a 3 TB/s card, streaming 140 GB of FP16 weights per decode step sets a hard per-step latency floor no amount of compute can beat. This is exactly why quantization helps decode so much: halving the bytes read nearly halves decode latency. It is also why batching is nearly free in decode, you re-read the same weights for many requests at once, and why disaggregating prefill and decode onto separate GPU pools is now standard for tight tail latency.
Continuous batching: fill the bubbles
Static batching groups N requests, runs them together, and returns when the slowest finishes. Because generation lengths vary wildly, short requests sit idle waiting for a long one to complete, and the GPU runs at a fraction of capacity. Worse, new arrivals wait for the entire batch to drain before they even start.
Continuous batching (also called in-flight or iteration-level batching) schedules at the granularity of a single decode step. After every iteration the scheduler evicts finished sequences and admits waiting ones, so the batch is recomposed continuously. A request that finishes at token 30 frees its slot immediately for a newcomer instead of stalling until token 2000.
Static batching optimizes a batch. Continuous batching optimizes the GPU. In production the difference is often 5-20x throughput at the same tail latency.
Paged attention and continuous batching are symbiotic: iteration-level scheduling only works if you can add and remove sequences of arbitrary length without pre-reserving contiguous KV. The block allocator makes that cheap, which is why vLLM ships both together.
Speculative decoding: buy back the bandwidth
Because decode is bandwidth-bound, a full weight read that produces one token is wasteful. Speculative decoding attacks this directly. A small, cheap draft model proposes k tokens; the large target model verifies all k in a single forward pass (one weight read), accepting the longest prefix that matches what it would have sampled. Verification is exact, so output quality is provably identical to the target model's own sampling distribution.
If the draft's acceptance rate is high, you get 2-3 tokens per target forward pass instead of one, a near-linear speedup on the bandwidth-bound path. The trade-off: draft misses waste compute, and self-speculation variants (Medusa, EAGLE, n-gram lookup) avoid running a separate model but add their own overhead. In practice speculation is a latency lever for interactive workloads, not a raw-throughput lever, since at high batch sizes the GPU is already saturated and there is no idle bandwidth to reclaim.
Quantization: fewer bytes, more tokens
Since decode is bandwidth-bound, the fastest way to speed it up is to move fewer bytes. Quantization shrinks weights (and optionally KV and activations) from FP16 to 8-bit or 4-bit. It cuts both the memory footprint, freeing HBM for a larger KV-cache and bigger batches, and the bytes streamed per step. The question is always accuracy versus throughput.
- INT8 / SmoothQuant: weights and activations to 8-bit; SmoothQuant migrates activation outliers into weights so per-tensor INT8 stays accurate. Roughly 2x memory cut, sub-1 percent quality loss on most tasks.
- FP8 (E4M3): native on Hopper/Ada tensor cores; better dynamic range than INT8 for activations, minimal calibration, near-lossless for weights and KV. Our default when the hardware supports it.
- GPTQ: 4-bit weight-only, second-order (Hessian-based) error compensation, calibrated post-training. Excellent compression, weight-only so activations stay FP16, some sensitivity on reasoning-heavy tasks.
- AWQ: 4-bit weight-only that protects the roughly 1 percent salient channels via per-channel scaling. Often better instruction-following retention than GPTQ at the same bit-width, with fast kernels.
- KV-cache quantization (INT8/FP8 KV): directly attacks the long-context problem, halving or quartering the cache footprint so 256K context becomes tractable.
The nuance experts miss: 4-bit weight-only quantization helps low-batch, latency-sensitive decode most, because there you are purely weight-bandwidth-bound. At very high batch sizes decode starts to look compute-bound again as KV reads and matmul work dominate, so INT8/FP8 with quantized activations and KV can win. There is no universal answer; we pick per model, per SLA, and validate accuracy on the customer's own eval set before it ships.
Tensor parallelism and the 256K problem
When a model plus its KV-cache exceeds one GPU, you shard. Tensor parallelism splits individual weight matrices across GPUs: each device holds a slice of every attention and MLP layer, computes its partial, and an all-reduce combines results, twice per transformer block. Because that all-reduce sits on the critical path of every layer, TP demands high-bandwidth interconnect (NVLink); stretched across PCIe or nodes, communication overhead destroys decode latency. TP also shards the KV-cache across devices, which is what makes long context fit at all.
Return to the 256K-context model. A single sequence needing ~78 GiB of KV cannot live on one card. Sharding KV across 8 GPUs via TP puts under 10 GiB per device, tractable alongside sharded weights. Layer on FP8 KV quantization to halve it again, prefix caching so shared context is not recomputed, and paged blocks so you only hold the tokens that exist. Long-context serving is not one trick; it is the entire stack composed. Pipeline parallelism can add cross-node scale, but its bubbles make it a throughput tool, not a latency one, so for interactive long-context we lean on TP within an NVLink island.
What actually decides cost-per-token and tail latency
Cost-per-token is throughput per GPU divided by the GPU's hourly cost. Everything above is a throughput lever: paged attention and continuous batching raise the batch you can sustain; quantization frees HBM for more KV and moves fewer bytes; TP makes otherwise-impossible models fit. Tail latency is the counter-force. Push batch size for throughput and per-step decode latency climbs, so p99 degrades. The engineering is finding the batch and the SLA where both are acceptable.
- Cost-per-token: maximize sustained tokens/sec/GPU (batch size, quantization, prefix cache hit rate, MFU).
- Median latency: prefill time (prompt length, compute-bound) plus per-token decode time (bandwidth-bound).
- Tail (p99) latency: scheduler fairness, preemption under load, prefill/decode contention, and how gracefully you shed or queue at saturation.
- The tension: the batch size that minimizes cost is rarely the one that minimizes p99, so serve to an explicit SLA, not a vibe.
# Live KV-cache footprint and how it caps concurrency.
# GQA 70B-class: 80 layers, 8 KV heads, head_dim 128.
def kv_bytes_per_token(layers, kv_heads, head_dim, dtype_bytes):
return 2 * layers * kv_heads * head_dim * dtype_bytes # K and V
per_tok_fp16 = kv_bytes_per_token(80, 8, 128, 2) # 320 KiB/token
per_tok_fp8 = kv_bytes_per_token(80, 8, 128, 1) # 160 KiB/token
ctx = 256_000
print(per_tok_fp16 * ctx / 2**30, "GiB for one 256K seq (FP16)") # ~78 GiB
print(per_tok_fp8 * ctx / 2**30, "GiB for one 256K seq (FP8)") # ~39 GiB
# KV budget left after weights sets max concurrent tokens in flight.
hbm, weights = 80 * 2**30, 40 * 2**30 # 80GB card, INT8-ish weights
kv_budget = hbm - weights
print(kv_budget // per_tok_fp8, "total cached tokens across all requests")None of this is magic; it is memory engineering. We do not rent AI, we engineer it, and at serving time that means owning the KV-cache math, the scheduler behavior, the quantization choice, and the parallelism topology end to end. On vLLM, Triton, and ONNX Runtime with our own orchestration, that is how we hit the cost-per-token and the p99 a production system actually requires.