Training cost was a count of operations. Serving cost is something else entirely: generating one token requires reading every weight in the model, which makes decoding a memory problem wearing a compute costume.
Ask what limits how fast a large model can write text, and the natural answer is arithmetic: it must do a great many multiplications per word. The natural answer is wrong by a factor of about a hundred and fifty. Generating a single token requires reading every weight in the model out of memory, and it is that reading, not the arithmetic, that sets the pace.
The reason is that serving splits into two regimes that behave nothing alike. Prefill handles the whole prompt at once: large matrices meet large matrices, and the arithmetic units are genuinely the constraint. Decode emits one token at a time, so a matrix meets a single vector, and the entire parameter store must stream out of memory to support a comparatively tiny amount of useful multiplication. That second regime is memory-bound, and nearly every serving optimization of recent years is a response to it.
The accelerator we will cost everything against
Throughout Part III, take a reference accelerator with 312 TFLOP/s of half-precision compute, 2 TB/s of memory bandwidth, and about 20 MB of fast on-chip memory. Half precision means 2 bytes per value. These are given constants, like a beam's Young's modulus — you never memorize them, you are handed them and you compute.
Attention at position \( t \) compares the current query against the keys of all previous positions and mixes their values. Recomputing those keys and values at every step would be quadratic waste, so they are stored — the KV cache. Its size is pure bookkeeping, and it is the first thing to compute about any serving setup.
The size of the cache
Per layer, per attention head that owns its own keys and values, and per sequence position, we store one key vector and one value vector, each of dimension \( d_{\text{head}} \). Hence for \( L \) layers, \( H_{kv} \) key/value heads, sequence length \( s \), batch \( B \), and \( \beta \) bytes per value:
\[ \text{cache bytes} = 2 \cdot L \cdot H_{kv} \cdot d_{\text{head}} \cdot s \cdot B \cdot \beta. \]The leading 2 counts keys and values separately — the single most commonly dropped factor in this calculation. Note that the cache is linear in sequence length: it is a per-token cost that accumulates, not a fixed overhead.
A cache that outgrows the model
Take a 7-billion-parameter model with \( L = 32 \) layers, 32 heads of dimension 128, half precision, at context \( s = 4096 \), batch 1.
Work the per-token figure first, since that is the quantity that accumulates. Per position the cache holds \( 2 \times 32 \times 32 \times 128 \times 2 = 524{,}288 \) bytes — about 0.5 MB per token. Over the full context: \( 524{,}288 \times 4096 = \) 2.15 GB.
The weights themselves are \( 7\times10^9 \times 2 = 14 \) GB. At batch 1 the cache is modest — but at batch 8 it becomes 17 GB and exceeds the weights. This, not arithmetic, is why long-context serving is expensive.
Now suppose the model shares keys and values across groups of heads, so that \( H_{kv} = 8 \) rather than 32. The cache falls by a factor of four, to 0.54 GB — the entire motivation for grouped-query attention.
A 70B model has \( L = 80 \) layers, \( H_{kv} = 8 \), \( d_{\text{head}} = 128 \), half precision. (a) Give the cache bytes per token. (b) At batch 4, what context length fills 40 GB?
(a) \( 2 \times 80 \times 8 \times 128 \times 2 = 327{,}680 \) bytes, about 0.33 MB per token. (b) \( 40\times10^9 / (4 \times 3.28\times10^5) \approx 30{,}500 \) tokens.
Whether a computation is limited by arithmetic or by memory is settled by comparing two ratios. The arithmetic intensity of a computation is the number of operations it performs per byte it moves. The ridge point of a machine is its peak operation rate divided by its bandwidth. Below the ridge, you are memory-bound; above it, compute-bound.
Try predicting the number first. On a machine that can do three hundred trillion operations per second, what fraction of that capacity do you suppose a large model uses while writing text one word at a time? Most people guess something like half, or a quarter if they are being pessimistic. The real figure is below one percent, and seeing exactly why will change how you read every performance claim in this field.
Why decode sits far below the ridge
Our reference machine has ridge point \( 312\times10^{12} / 2\times10^{12} = \) 156 operations per byte.
A large square matrix multiply performs \( n^3 \) operations while moving about \( 3n^2 \) values, an intensity of roughly \( n/3 \) — for large \( n \) this sits comfortably above the ridge, so prefill is compute-bound.
Batch-1 decode is the opposite. Producing one token performs about \( 2N \) operations (one multiply–add per parameter) while moving about \( 2N \) bytes (reading every parameter once in half precision). The intensity is about 1 operation per byte — a factor of 156 below the ridge. Decoding uses well under one percent of the machine's arithmetic capability.
Batching \( B \) requests reuses each weight read \( B \) times, so the intensity rises to roughly \( B \). You approach compute-bound only near \( B \approx 156 \) — which is precisely why serving systems are obsessed with batch size.
The decode speed limit
A 7B model in half precision holds 14 GB of weights, all of which must be read to emit one token. The ceiling is therefore
\[ \frac{2\text{ TB/s}}{14\text{ GB}} = \frac{2\times10^{12}}{1.4\times10^{10}} \approx \textbf{143 tokens per second}, \]no matter how fast the arithmetic units are. Doubling the machine's FLOP rate changes nothing; halving the bytes per weight doubles the ceiling. This single division reframes what “a faster model” means.
Two errors to guard against. First, cache figures are often quoted per token and often quoted as totals, and the two differ by a factor of the context length; always state which you mean. Second — and this is a habit for all of Part III — every efficiency claim must name the budget it improves. Grouped-query attention shrinks the cache, not the weight read, so it helps long-context and large-batch serving but does not raise the batch-1 decode ceiling. “It makes it faster” without naming a budget is not an answer.
A final piece of accounting: what fraction of a machine's capability is a workload actually using? For training, the model FLOP utilization is the useful operations per second divided by peak — with \( 6N \) operations per token from Chapter 7,
\[ \text{MFU} = \frac{\text{tokens/s} \times 6N}{\text{peak FLOP/s}}. \]A well-tuned training run reaches a substantial fraction. Decode, as Derivation 11.2 showed, cannot: its utilization is structurally terrible, because it is not limited by the resource the denominator measures. Reporting decode MFU without saying so is a way of making a memory-bound workload look like an engineering failure rather than a physical one.
A training run sustains 4200 tokens/s per accelerator on a 7B model. What is its MFU on the reference machine?
\( 4200 \times 6 \times 7\times10^9 = 1.76\times10^{14} \) FLOP/s \( = 176 \) TFLOP/s. Divided by 312: 56%.
You can now price a deployed model with three formulas, and none of them counts a floating-point operation. The cache size, linear in context and often larger than the weights. The roofline, which places batch-1 decode a hundred and fifty times below the point where arithmetic matters. And the decode ceiling, bandwidth divided by weight bytes, which says what a model can possibly emit per second. Together they explain why the modern era's optimizations target bytes rather than operations — and Chapter 12 takes up the two most elegant examples: an attention kernel that does more arithmetic to move less memory, and a decoding scheme that buys parallelism from a cheap draft model while provably changing nothing about the output.
A · Drills
B · Problems
C · Challenge
Reproduce Derivations 10.1 (cache size, with the factor of 2 and the right head count) and 10.2 (intensities and the ridge point) on blank paper, and produce the 143-tokens-per-second ceiling from nothing. Attempt B-1 and B-3 closed book. You pass when you instinctively ask “which budget?” of every optimization claim.