Once architecture stops being the bottleneck, progress becomes a budgeting problem: given a fixed amount of compute, how should you split it between a bigger model and more data? The answer is a single constrained minimization — the most consequential in the field.
You have a fixed budget of computation, and two ways to spend it: a bigger model, or more data for a smaller one. Spend it wrong and you will train a model that is worse than the one the same money could have bought. For years this was decided by intuition and precedent. It is a calculus problem, and the calculus is the most valuable thing in this book.
What makes it tractable is that loss falls as a smooth power law in both model size and data, and that compute is very nearly the product of the two. Those two facts are enough. We will set the problem up, solve it by substitution, read off how each quantity should scale with the budget, and finish by re-budgeting a famously over-large model into a much smaller and better-trained one at identical cost.
Empirically, the loss of a trained language model is captured remarkably well by a sum of three terms:
\[ L(N, D) = A N^{-a} + B D^{-b} + E. \]Here \( N \) is the number of parameters and \( D \) the number of training tokens. The first term falls as the model grows; the second falls as the data grows; the third, \( E \), is an irreducible floor — the entropy of language itself, which no amount of scale removes. The exponents \( a \) and \( b \) are small positive numbers, measured from experiments and, in the spirit of this book, given to you when needed. The shape of this surface is all we require.
Two facts about power laws set the stage. First, because the exponents are small, improvements are expensive: halving a term takes far more than a doubling of its variable. Second, the two reducible terms trade off against each other under a compute budget, and finding the best trade is the whole game.
What a halving really costs
Take the parameter term with exponent \( a = 0.076 \). To halve it we need \( N^{-0.076} \) to fall by a factor of two, i.e. to multiply \( N \) by \( k \) where \( k^{-0.076} = \tfrac12 \). Then \( 0.076 \log_2 k = 1 \), so \( \log_2 k = 13.16 \) and \( k = 2^{13.16} \approx 9{,}100 \). Halving this loss term demands a nine-thousand-fold increase in parameters. The power law is smooth and utterly dependable — and staggeringly expensive. “Reliable” is not the same as “cheap.”
Training compute is, to a good approximation, proportional to the product of parameters and tokens: \( C = kND \), with \( k = 6 \) — two floating-point operations per parameter per token on the forward pass and four on the backward, as counted in Chapter 2. It is not an empirical constant; it is three passes over the weights at two operations each. Given a fixed budget \( C \), we want the split of that budget between \( N \) and \( D \) that minimizes the loss. This is a constrained minimization, and we solve it exactly as Chapter 1 prescribed: use the constraint to eliminate one variable, then differentiate.
One warning before we start, because nearly everyone makes the same mistake here on first contact. When two terms trade off under a constraint, it is tempting to assume the optimum is where the two terms are equal. That is a good instinct and it is very nearly right — but not quite, and the correction matters. Watch for where the exponents appear, and note that they appear because we differentiated, not because anyone put them there by hand.
The exponent-weighted balance
Minimize \( L(N,D) = AN^{-a} + BD^{-b} + E \) subject to \( C = kND \). Substitute \( D = C/(kN) \), so the data term becomes \( B\,(kN/C)^{b} \), and the irreducible \( E \) drops out of the differentiation:
\[ L(N) = A N^{-a} + B\left(\tfrac{k}{C}\right)^{b} N^{b}. \]Differentiate and set to zero:
\[ \frac{dL}{dN} = -aA N^{-a-1} + bB\left(\tfrac{k}{C}\right)^{b} N^{b-1} = 0. \]The second term is \( bB D^{-b}/N \) once we substitute \( D \) back. Multiplying through by \( N \) gives the optimum condition:
\[ \boxed{\,aA N^{-a} = bB D^{-b}\,}. \]At the optimal allocation, the two reducible loss terms are equal after weighting each by its own exponent. Solving the balance together with \( ND \propto C \) yields the scaling of each with the budget:
\[ N \propto C^{\,b/(a+b)}, \qquad D \propto C^{\,a/(a+b)}. \]And the ratio of tokens to parameters scales as \( D/N \propto C^{(a-b)/(a+b)} \).
It is tempting to write the balance as \( AN^{-a} = BD^{-b} \) — the bare terms equal, without the exponent weights. This is wrong, and it produces the wrong optimal ratio whenever \( a \neq b \). The reason the exponents appear is that the constraint trades \( N \) for \( D \) at a rate that depends on how fast each term responds — and that rate is exactly the exponent. Substitution makes the weights fall out automatically, which is precisely why we solve it that way rather than by guessing that the terms should balance.
The special case \( a = b \) is worth naming, because it is the regime people carry in their heads. When the exponents are equal, \( N \propto C^{1/2} \) and \( D \propto C^{1/2} \): parameters and tokens grow together as the square root of compute, and the ratio \( D/N \) is a constant, independent of budget. “Use a fixed number of tokens per parameter” — the rule of thumb that a model should see roughly twenty tokens for each parameter — is this equal-exponent case.
Show that when \( a = b \), the token-to-parameter ratio \( D/N \) is independent of the compute budget \( C \), and that both \( N \) and \( D \) scale as \( C^{1/2} \).
With \( a = b \), the exponents in \( N \propto C^{b/(a+b)} \) and \( D \propto C^{a/(a+b)} \) both equal \( \tfrac12 \). Then \( D/N \propto C^{1/2}/C^{1/2} = C^0 \), a constant.
The derivation's practical bite is that it exposes over-large, under-trained models. A model that is too big for the data it saw is spending compute inefficiently: the same compute, re-allocated toward the optimal ratio, would train a smaller model on more data to a lower loss.
From 175 billion to 51 billion
Consider a well-known model of \( N = 175 \) billion parameters trained on \( D = 300 \) billion tokens — a ratio of under two tokens per parameter, far below the optimal twenty. Its compute is \( C = 6ND = 6 \times (1.75\times 10^{11})(3\times 10^{11}) = 3.15\times 10^{23} \) operations.
Hold that compute fixed and re-allocate to the ratio \( D = 20N \). Then \( C = 6N(20N) = 120 N^2 \), so \( N^2 = 3.15\times 10^{23}/120 = 2.625\times 10^{21} \), giving \( N = 5.12\times 10^{10} \) — about 51 billion parameters, trained on roughly \( 1.02 \) trillion tokens. The same compute buys a model a third the size, trained on more than three times the data, at a lower loss. This single calculation redirected how the field spends its largest training budgets.
Under the coupling \( D \propto N^{0.74} \), if parameters are increased by a factor of 1000, by what factor should the dataset grow?
\( 1000^{0.74} = 10^{3 \times 0.74} = 10^{2.22} \approx 166 \). A thousandfold in parameters calls for about a 166-fold increase in data.
The compute-optimal allocation is a constrained minimization whose answer — the exponent-weighted balance and the resulting \( C^{b/(a+b)} \) scaling — turned a matter of intuition into a matter of arithmetic. Its equal-exponent special case is the familiar “fixed tokens per parameter” rule; its general form exposes when a model is misallocated; and applying it to a real over-large model shrinks it by a factor of three at equal compute. This is the derivation to own most thoroughly in the entire book, and the three-days protocol below is not optional. Chapter 8 turns from the economics of scale to its machinery: once a model is too large for one accelerator, how is the work split across many, and what does that splitting cost?
A · Drills
B · Problems
C · Challenge
Reproduce Derivation 7.1 — substitution, differentiation, the exponent-weighted balance, and the \( C^{b/(a+b)} \) scalings — on blank paper, from nothing, on three separate days. Between attempts, do not reread it; let it settle and reconstruct it cold. This is the second of the book's three protocol derivations, and it is the one you will lean on most: Chapter 14's DPO collapse uses the same substitute-and-cancel discipline, and the fluency transfers directly. You pass when all three reconstructions are complete and correct, and B-2's re-budgeting comes without notes.