← The Arithmetic of Intelligence
Part III · The Modern Era · Chapter 17

The Whole Lifecycle

Chapter 7 minimized the cost of building a model. A deployed model also costs something every time it answers — and once that is in the objective, the optimal model changes.

We opened, in Chapter 7, by minimizing what it costs to build a model. That was only half the bill. A deployed model charges you again every time it answers, for as long as it is in service — and once that second charge is in the objective, the model you should have built is not the one Chapter 7 told you to build.

17.1 The objective with serving in it

Training a model of \( N \) parameters on \( D \) tokens costs about \( 6ND \) operations (Chapter 7). Serving it costs about \( 2N \) operations per generated token — one multiply–add per parameter, forward only. Over a deployment lifetime of \( T \) tokens, total cost is

\[ C_{\text{total}} = \underbrace{6 N D_{\text{train}}}_{\text{build}} + \underbrace{2 N T}_{\text{serve}}. \]

Look at the second term before going further, because everything in this chapter follows from its shape. It is proportional to \( N \) alone. So if two models reach the same quality by different routes — one large and briefly trained, one smaller and trained far longer — the smaller one is cheaper to serve forever, and as \( T \) grows that advantage eventually dominates whatever extra it cost to build.

Worked Example 17.1

The smaller model wins the long run

Take two options of equal quality (this equivalence is given; establishing it empirically is the hard part).

Option A: \( N = 70 \) B, \( D = 1.4 \) T. Option B: \( N = 35 \) B, \( D = 4.2 \) T.

At a lifetime of \( T = 10^{13} \) tokens:

\( C_A = 6(7{\times}10^{10})(1.4{\times}10^{12}) + 2(7{\times}10^{10})(10^{13}) = 5.88{\times}10^{23} + 1.40{\times}10^{24} = \) 1.99×1024.

\( C_B = 6(3.5{\times}10^{10})(4.2{\times}10^{12}) + 2(3.5{\times}10^{10})(10^{13}) = 8.82{\times}10^{23} + 7.00{\times}10^{23} = \) 1.58×1024.

Option B costs 50% more to train and still wins the lifetime by about 20% — and every serving number from Chapter 11 (cache size, decode ceiling, memory footprint) improves at 35 B as well. The compute-optimal recipe of Chapter 7 was answering a different question than the one a deploying organization actually faces.

Derivation 17.1

The break-even lifetime

Two iso-quality options tie when their total costs are equal:

\[ 6(ND)_A + 2N_A T = 6(ND)_B + 2N_B T \ \Longrightarrow\ T^* = \frac{6\bigl[(ND)_A - (ND)_B\bigr]}{2\,(N_B - N_A)}. \]

For the example above: \( (ND)_A = 9.8\times10^{22} \), \( (ND)_B = 1.47\times10^{23} \), so the numerator is \( 6(-4.9\times10^{22}) \); with \( N_B - N_A = -3.5\times10^{10} \), we get \( T^* = \frac{2.94\times10^{23}}{7.0\times10^{10}} = \) 4.2×1012 tokens. Below that lifetime the larger model is cheaper overall; above it, the smaller one. The right model to train is a function of expected demand.

17.2 But the optimum is interior

If smaller-and-longer-trained is better, why not push it arbitrarily far? Because iso-quality has diminishing returns: matching a given quality with an ever-smaller model demands disproportionately more data, and the training term eventually overwhelms the serving saving.

Worked Example 17.2

Pushing too far

Add Option C: \( N = 20 \) B, \( D = 12 \) T, also of equal quality. At \( T = 10^{13} \):

\( C_C = 6(2{\times}10^{10})(1.2{\times}10^{13}) + 2(2{\times}10^{10})(10^{13}) = 1.44{\times}10^{24} + 4.00{\times}10^{23} = \) 1.84×1024.

Worse than Option B's \( 1.58\times10^{24} \), though still better than A. The optimum lies between A and C — an interior point, not a corner. This is why the answer is a calculation rather than a slogan: “train smaller models longer” is directionally right and quantitatively unbounded, and only the arithmetic locates the stopping point.

Drill 17.1

At a lifetime of only \( T = 10^{11} \) tokens, recompute the totals for Options A and B above and say which wins.

Show answer

\( C_A = 5.88{\times}10^{23} + 2(7{\times}10^{10})(10^{11}) = 5.88{\times}10^{23} + 1.4{\times}10^{22} = 6.02{\times}10^{23} \). \( C_B = 8.82{\times}10^{23} + 7{\times}10^{21} = 8.89{\times}10^{23} \). A wins — below the break-even lifetime, the cheaper-to-train model is correct.

17.3 Distillation

A second lever changes the iso-quality menu itself. Train a large, expensive teacher once; then train a small student to imitate not just the teacher's answers but its full output distribution. The soft distribution carries more information per example than a hard label — a statement Chapter 10 lets us make precisely, since a distribution over outcomes specifies more bits than a single outcome does. The student reaches a quality its own size and data would not otherwise buy, and it is the student that gets served.

The modern refinement grades the student's own samples rather than only imitating the teacher's, which connects this chapter directly to Chapters 14 and 15: the teacher (or a verifier) supplies the reward signal, and the student improves on the distribution it actually produces.

17.4 The four budgets

The book has priced four distinct scarce resources, and naming them is the last piece of vocabulary to carry away.

Nearly every technique in Part III is an arbitrage between two of these: spend an abundant budget to relieve a binding one. Streaming softmax spends arithmetic to save bandwidth. Speculation spends draft compute to save weight reads. Quantization spends precision to save bytes. Low-rank adaptation spends expressiveness to save optimizer state. Mixture-of-experts spends memory to save per-token compute. Distillation spends a large training run once to save serving forever. Once you see the four budgets, the field's papers stop being a list of tricks and become a small set of trades.

17.5 What the book bought

You began with six mathematical reflexes and have ended able to derive, from nothing: why deep networks were hard to train and what fixed it; why residual connections work, three separate ways; how memory decays in a recurrence and what pins it; why attention scales its scores and what it costs; how to allocate a compute budget optimally, and what a halving of loss really costs; what pipelining and sharding cost; how a benchmark can lie by threshold or by leakage; why compression is the master currency and Occam a theorem; why serving is a memory problem; how two exact techniques buy speed for free; what sparsity and quantization and adaptation each save; how a reinforcement-learning pipeline collapsed into a supervised loss; what sampling more answers can and cannot achieve; how noise-reversal generates images; and how the whole lifecycle changes what you should have built.

None of it required memorizing a benchmark score or a date. The thirty-eight derivations of Appendix A are the entire syllabus, and the trap taxonomy of Appendix B names the ways they are misapplied. The real examination is not the mock paper. It is next quarter's arXiv posting: read a page, name the budget it arbitrages, and identify the derivation that prices it. If you can do that, the program worked — and unlike any particular result in this book, that skill does not expire.

Exercises

A · Drills

  1. A 40 B model serves \( 5\times10^{12} \) tokens. Give the serving cost in operations.
  2. The same model was trained on 2 T tokens. Give the training cost, and the ratio of serving to training.
  3. State the break-even formula for two iso-quality options.
  4. Name the four budgets and give one technique that relieves each.

B · Problems

  1. A deployment decision. Two iso-quality options: P has \( N = 100 \) B, \( D = 2 \) T; Q has \( N = 40 \) B, \( D = 6 \) T. Compute total lifetime cost at \( T = 10^{12} \) and \( T = 10^{14} \), find the break-even lifetime, and recommend one, stating your assumption about demand.
  2. Where the interior optimum sits. Given iso-quality options at \( (N, D) \) of (80 B, 1 T), (40 B, 3 T), (20 B, 10 T), and (10 B, 40 T), compute lifetime cost at \( T = 5\times10^{12} \) for each and identify the best. Explain why both extremes lose.
  3. Serving beyond FLOPs. Worked Example 17.1 counts only operations. Using Chapter 11, list three further ways the 35 B option beats the 70 B option in production, quantifying at least one of them. Then state why an operations-only account understates the case for the smaller model.

C · Challenge

  1. The unified objective. Write an objective that, minimized, would jointly determine the model size, training tokens, and quantity of preference data for a deployment with known lifetime \( T \) and known per-unit costs of compute and human annotation. State what empirical inputs it needs that this book has treated as given, and identify which of them is hardest to obtain and why. (You are being asked to state honestly where derivation ends and measurement must begin — the most valuable judgment in the field.)
Gate 17 · The final gate

Reproduce Derivation 17.1 and both worked examples on blank paper. Then sweep the entire Derivation Bank — all thirty-eight results, Chapters 3 through 17 — reproducing each from nothing over a few sittings. Finally, take the mixed examination described in Appendix D's preface under timed conditions. You pass the book, not merely the chapter, when the sweep is complete and the trap taxonomy in Appendix B contains no class you have not personally fallen into and logged.

Readings for Chapter 17