Chapter 7 minimized the cost of building a model. A deployed model also costs something every time it answers — and once that is in the objective, the optimal model changes.
We opened, in Chapter 7, by minimizing what it costs to build a model. That was only half the bill. A deployed model charges you again every time it answers, for as long as it is in service — and once that second charge is in the objective, the model you should have built is not the one Chapter 7 told you to build.
Training a model of \( N \) parameters on \( D \) tokens costs about \( 6ND \) operations (Chapter 7). Serving it costs about \( 2N \) operations per generated token — one multiply–add per parameter, forward only. Over a deployment lifetime of \( T \) tokens, total cost is
\[ C_{\text{total}} = \underbrace{6 N D_{\text{train}}}_{\text{build}} + \underbrace{2 N T}_{\text{serve}}. \]Look at the second term before going further, because everything in this chapter follows from its shape. It is proportional to \( N \) alone. So if two models reach the same quality by different routes — one large and briefly trained, one smaller and trained far longer — the smaller one is cheaper to serve forever, and as \( T \) grows that advantage eventually dominates whatever extra it cost to build.
The smaller model wins the long run
Take two options of equal quality (this equivalence is given; establishing it empirically is the hard part).
Option A: \( N = 70 \) B, \( D = 1.4 \) T. Option B: \( N = 35 \) B, \( D = 4.2 \) T.
At a lifetime of \( T = 10^{13} \) tokens:
\( C_A = 6(7{\times}10^{10})(1.4{\times}10^{12}) + 2(7{\times}10^{10})(10^{13}) = 5.88{\times}10^{23} + 1.40{\times}10^{24} = \) 1.99×1024.
\( C_B = 6(3.5{\times}10^{10})(4.2{\times}10^{12}) + 2(3.5{\times}10^{10})(10^{13}) = 8.82{\times}10^{23} + 7.00{\times}10^{23} = \) 1.58×1024.
Option B costs 50% more to train and still wins the lifetime by about 20% — and every serving number from Chapter 11 (cache size, decode ceiling, memory footprint) improves at 35 B as well. The compute-optimal recipe of Chapter 7 was answering a different question than the one a deploying organization actually faces.
The break-even lifetime
Two iso-quality options tie when their total costs are equal:
\[ 6(ND)_A + 2N_A T = 6(ND)_B + 2N_B T \ \Longrightarrow\ T^* = \frac{6\bigl[(ND)_A - (ND)_B\bigr]}{2\,(N_B - N_A)}. \]For the example above: \( (ND)_A = 9.8\times10^{22} \), \( (ND)_B = 1.47\times10^{23} \), so the numerator is \( 6(-4.9\times10^{22}) \); with \( N_B - N_A = -3.5\times10^{10} \), we get \( T^* = \frac{2.94\times10^{23}}{7.0\times10^{10}} = \) 4.2×1012 tokens. Below that lifetime the larger model is cheaper overall; above it, the smaller one. The right model to train is a function of expected demand.
If smaller-and-longer-trained is better, why not push it arbitrarily far? Because iso-quality has diminishing returns: matching a given quality with an ever-smaller model demands disproportionately more data, and the training term eventually overwhelms the serving saving.
Pushing too far
Add Option C: \( N = 20 \) B, \( D = 12 \) T, also of equal quality. At \( T = 10^{13} \):
\( C_C = 6(2{\times}10^{10})(1.2{\times}10^{13}) + 2(2{\times}10^{10})(10^{13}) = 1.44{\times}10^{24} + 4.00{\times}10^{23} = \) 1.84×1024.
Worse than Option B's \( 1.58\times10^{24} \), though still better than A. The optimum lies between A and C — an interior point, not a corner. This is why the answer is a calculation rather than a slogan: “train smaller models longer” is directionally right and quantitatively unbounded, and only the arithmetic locates the stopping point.
At a lifetime of only \( T = 10^{11} \) tokens, recompute the totals for Options A and B above and say which wins.
\( C_A = 5.88{\times}10^{23} + 2(7{\times}10^{10})(10^{11}) = 5.88{\times}10^{23} + 1.4{\times}10^{22} = 6.02{\times}10^{23} \). \( C_B = 8.82{\times}10^{23} + 7{\times}10^{21} = 8.89{\times}10^{23} \). A wins — below the break-even lifetime, the cheaper-to-train model is correct.
A second lever changes the iso-quality menu itself. Train a large, expensive teacher once; then train a small student to imitate not just the teacher's answers but its full output distribution. The soft distribution carries more information per example than a hard label — a statement Chapter 10 lets us make precisely, since a distribution over outcomes specifies more bits than a single outcome does. The student reaches a quality its own size and data would not otherwise buy, and it is the student that gets served.
The modern refinement grades the student's own samples rather than only imitating the teacher's, which connects this chapter directly to Chapters 14 and 15: the teacher (or a verifier) supplies the reward signal, and the student improves on the distribution it actually produces.
The book has priced four distinct scarce resources, and naming them is the last piece of vocabulary to carry away.
Nearly every technique in Part III is an arbitrage between two of these: spend an abundant budget to relieve a binding one. Streaming softmax spends arithmetic to save bandwidth. Speculation spends draft compute to save weight reads. Quantization spends precision to save bytes. Low-rank adaptation spends expressiveness to save optimizer state. Mixture-of-experts spends memory to save per-token compute. Distillation spends a large training run once to save serving forever. Once you see the four budgets, the field's papers stop being a list of tricks and become a small set of trades.
You began with six mathematical reflexes and have ended able to derive, from nothing: why deep networks were hard to train and what fixed it; why residual connections work, three separate ways; how memory decays in a recurrence and what pins it; why attention scales its scores and what it costs; how to allocate a compute budget optimally, and what a halving of loss really costs; what pipelining and sharding cost; how a benchmark can lie by threshold or by leakage; why compression is the master currency and Occam a theorem; why serving is a memory problem; how two exact techniques buy speed for free; what sparsity and quantization and adaptation each save; how a reinforcement-learning pipeline collapsed into a supervised loss; what sampling more answers can and cannot achieve; how noise-reversal generates images; and how the whole lifecycle changes what you should have built.
None of it required memorizing a benchmark score or a date. The thirty-eight derivations of Appendix A are the entire syllabus, and the trap taxonomy of Appendix B names the ways they are misapplied. The real examination is not the mock paper. It is next quarter's arXiv posting: read a page, name the budget it arbitrages, and identify the derivation that prices it. If you can do that, the program worked — and unlike any particular result in this book, that skill does not expire.
A · Drills
B · Problems
C · Challenge
Reproduce Derivation 17.1 and both worked examples on blank paper. Then sweep the entire Derivation Bank — all thirty-eight results, Chapters 3 through 17 — reproducing each from nothing over a few sittings. Finally, take the mixed examination described in Appendix D's preface under timed conditions. You pass the book, not merely the chapter, when the sweep is complete and the trap taxonomy in Appendix B contains no class you have not personally fallen into and logged.