Three ways to make a model cheaper: use only part of it per token, store it in fewer bits, or train only a thin slice of it. Each attacks a different budget, and saying which is the whole skill.
Three techniques in this chapter all get described with the same word: cheaper. Mixture-of-experts is cheaper. Quantization is cheaper. Low-rank adaptation is cheaper. They are cheaper in three entirely different currencies, and a great deal of professional confusion comes from not saying which. Before we begin, then, a rule: no claim of efficiency counts until it names the resource it saves.
If you take one habit from this chapter, take this one. Fix the vocabulary before anything else. Chapter 17 will name four budgets for the whole book — training operations, serving operations, memory bytes and bandwidth, and human preference information. This chapter works inside the third of them and splits it three ways: the weights themselves; the gradients and optimizer state that accompany any trainable weight, itemised in Derivation 8.2; and the activations held for the backward pass, priced in Section 8.3. Set beside per-token computation — a slice of the first two budgets rather than of memory — that gives the four resources in play here. Each technique below improves exactly one of them and leaves the rest untouched — and mistaking which is trap class T9, which is expensive in both senses of the word.
A dense layer spends every parameter on every token. A mixture-of-experts layer instead holds \( E \) parallel expert sub-networks and a small router that sends each token to the top \( k \) of them. Total parameters scale with \( E \); per-token computation scales with \( k \). The model's “size” splits into two numbers, and every claim about such a model must say which it means.
Total versus active
Take 32 layers; in each, attention costs 50 M parameters, and there are 8 experts of 150 M each with top-2 routing.
Total: \( 32\,(50 + 8\times150)\text{ M} = 32 \times 1250\text{ M} = \) 40 B parameters.
Active per token: \( 32\,(50 + 2\times150)\text{ M} = 32 \times 350\text{ M} = \) 11.2 B parameters.
So training compute (\( 6 N_{\text{active}} D \), Chapter 7) and decode arithmetic behave like an 11-billion-parameter model, while memory — every expert must be resident — behaves like a 40-billion one. A mixture-of-experts buys capacity at small-model compute prices while paying large-model rent.
The router must spread tokens evenly, or a few experts receive everything while others starve and never train. Balance is encouraged by an auxiliary loss. Writing \( f_i \) for the fraction of tokens routed to expert \( i \) and \( P_i \) for the mean router probability assigned to it,
\[ \mathcal{L}_{\text{aux}} = E \sum_{i=1}^{E} f_i P_i, \]which equals 1 under perfect balance and exceeds 1 otherwise. It is a pressure, not a guarantee. The hard backstop is expert capacity: each expert accepts at most \( \text{(capacity factor)} \times \text{tokens}/E \) tokens per batch, and tokens beyond that are dropped — a quality cliff that no loss curve displays directly.
Imbalance, and what it costs
Two experts receive fractions \( f = (0.7, 0.3) \) with mean router probabilities \( P = (0.6, 0.4) \). Then \( \mathcal{L}_{\text{aux}} = 2\,(0.7 \times 0.6 + 0.3 \times 0.4) = 2\,(0.42 + 0.12) = 1.08 \), above the balanced minimum of 1.
Now capacity. With 1024 tokens, 8 experts, and a capacity factor of 1.25, each expert accepts \( 1.25 \times 1024/8 = 160 \) tokens. If 200 tokens are routed to expert 3, then 40 are dropped — they pass through without that expert's contribution.
A model has 48 layers; each has 60 M shared parameters and 64 experts of 40 M, with top-2 routing. Give total and active parameter counts and their ratio.
Total \( = 48(60 + 64\times40)\text{M} = 48 \times 2620\text{M} = 125.8 \) B. Active \( = 48(60 + 2\times40)\text{M} = 48 \times 140\text{M} = 6.7 \) B. Ratio \( \approx 18.7 \).
Storing weights in \( b \) bits instead of 16 shrinks the weight budget proportionally — and, by Chapter 11's ceiling, speeds up memory-bound decoding by the same factor. That second consequence surprises people, so it is worth stating plainly: on a decode workload, quantization is a speedup, because the ceiling is bandwidth divided by bytes read.
The cost is precision. A uniform quantizer covering \( [-R, R] \) with \( b \) bits has step size \( \Delta = 2R/2^{b} \), and the resulting error, modeled as uniform over one step, has variance
\[ \text{MSE} = \frac{\Delta^2}{12}. \]The quantization error, and two sensitivities
A value uniformly distributed on \( [-\Delta/2, \Delta/2] \) has variance \( \frac{1}{\Delta}\int_{-\Delta/2}^{\Delta/2} x^2\,dx = \frac{\Delta^2}{12} \). Two consequences follow immediately from \( \Delta = 2R/2^b \):
Each extra bit quarters the error. Doubling \( b \) by one halves \( \Delta \), and the error goes as \( \Delta^2 \).
The error grows as the square of the range. If a few outlier weights force \( R \) from 1 to 4, the step quadruples and the MSE grows sixteenfold — at unchanged bit width. This is precisely why a handful of outlier channels wreck naive low-bit quantization, and why practical methods handle those channels separately.
Which weights deserve the bits is itself the question Chapter 10 answered in principle: precision costs, so spend it where it buys accuracy. Modern quantization methods are, in this light, allocation schemes over a bit budget — the Hinton–van Camp argument with an engineering budget attached.
(a) How many bits keep the MSE at or below \( 10^{-4} \) over the range \( [-1,1] \)? (b) A 70B model at 16, 8, and 4 bits: give the weight memory and the batch-1 decode ceiling on the reference machine.
(a) \( \Delta \le \sqrt{12\times10^{-4}} = 0.0346 \); \( 2/2^b \le 0.0346 \Rightarrow 2^b \ge 57.7 \Rightarrow b = 6 \). (b) 140 / 70 / 35 GB; ceilings \( 2000/140 = 14.3 \), \( 28.6 \), \( 57 \) tokens/s.
The third technique changes what is trainable. Freeze a weight matrix \( W \in \mathbb{R}^{d\times d} \) and learn only a low-rank update \( \Delta W = BA \), with \( B \in \mathbb{R}^{d\times r} \) and \( A \in \mathbb{R}^{r\times d} \) for a small rank \( r \). The trainable parameter count falls from \( d^2 \) to \( 2rd \).
Which budget this actually attacks
For \( d = 4096 \) and \( r = 16 \): trainable parameters \( 2 \times 16 \times 4096 = 131{,}072 \) against \( d^2 = 16.78 \) M — 0.78% of the matrix.
Now the point. The sixteen bytes per parameter of Derivation 8.2 are paid only on trainable parameters — fourteen of them, strictly, since the frozen weight must still be read. Fine-tuning a 7-billion-parameter model in full would demand about 112 GB of optimizer state; adapting roughly one percent of it demands about 1.1 GB. That is why such fine-tuning fits on a single accelerator — and it has essentially nothing to do with arithmetic, since the forward and backward passes still traverse the full frozen model.
Three near-universal misstatements. Quantization does not slow decoding down by adding work — it speeds it up, because decode is bandwidth-bound. Low-rank adaptation does not make training arithmetic cheaper — it makes optimizer memory cheaper, which then buys larger batches or smaller hardware. And a mixture-of-experts with the same active parameters as a dense model has the same compute but far greater memory. In each case the claim is only meaningful once you name the budget.
At \( d = 8192 \), adapting all four attention projection matrices at rank \( r = 8 \): give the total trainable parameters and the percentage of the \( 4d^2 \) attention weights.
\( 4 \times 2 \times 8 \times 8192 = 524{,}288 \). Attention weights \( 4d^2 = 268.4 \) M. Fraction \( = 0.20\% \).
Each technique here saves something real, and no two of them save the same thing. Mixture-of-experts decouples capacity from per-token compute, at the price of resident memory and a routing problem whose failure mode is silently dropped tokens. Quantization shrinks weights, and because decode is bandwidth-bound it buys speed directly — with an error that quarters per bit and grows as the square of the range. Low-rank adaptation shrinks optimizer state by two orders of magnitude, which is what made fine-tuning broadly accessible. Chapter 14 uses that accessibility: it is the chapter where a pretrained model is taught what kind of answer is wanted, and it contains the last of the book's three protocol derivations.
A · Drills
B · Problems
C · Challenge
Reproduce the active-versus-total accounting, Derivation 13.1 (the \( \Delta^2/12 \) error with both sensitivities), and the \( 2rd \) count with its budget argument, on blank paper. Attempt B-1 and B-3 closed book. You pass when, for any efficiency claim, you name its budget before evaluating it.