Solutions

\( 0.9^{100} = e^{100\ln 0.9} = e^{-10.54} = 2.7\times10^{-5} \). \( 0.99^{100} = e^{-1.005} = 0.366 \). The first design annihilates the signal; the second preserves over a third of it. The lesson is the sensitivity of a compounded factor: a change of \( 0.09 \) in the per-step multiplier changes the hundred-step survival by four orders of magnitude.

Independent variances add, so the accumulated variance is \( 0.01L \) and the standard deviation is \( 0.1\sqrt{L} \). At \( L = 100 \), it is \( 1.0 \) — comparable to a unit-scale signal.

Substitute \( D = C/N \) into \( f = AN^{-a} + BD^{-b} \) to get \( f(N) = AN^{-a} + B C^{-b} N^{b} \). Then \( f'(N) = -aAN^{-a-1} + bBC^{-b}N^{b-1} = 0 \). Multiply by \( N \) and recognize \( BC^{-b}N^{b} = BD^{-b} \): the condition is \( aAN^{-a} = bBD^{-b} \). The bare terms are equal only if \( a = b \), because the constraint exchanges \( N \) for \( D \) at a rate set by how fast each term responds — and that rate is the exponent.

\( 0.7 = c + 0.6(1-c) = 0.6 + 0.4c \Rightarrow c = 0.25 \). By Bayes, \( P(\text{contaminated}\mid\text{correct}) = c/0.7 = 0.357 \). Over a third of the successes are recall.

Halving requires \( k^{-\alpha} = \tfrac12 \), so \( k = 2^{1/\alpha} \). At \( \alpha = 0.076 \): \( 1/\alpha = 13.16 \) and \( k = 2^{13.16} \approx 9{,}100 \). Separately, \( 1000^{0.74} = 10^{2.22} \approx 166 \). Together: a nine-thousand-fold parameter increase for one halving, with data required to grow 166-fold per thousandfold of parameters. Progress by scale alone is dependable and brutally priced — which is why the allocation question of Chapter 7 matters so much.

Tables: \( 2 \times 32{,}000 \times 1024 = 65.54 \) M. Each hidden layer holds \( 1024 \times 1024 + 1024 = 1{,}049{,}600 \), so twelve hold \( 12.60 \) M. Total \( 78.13 \) M, of which the tables are \( 65.54/78.13 = 84\% \).

The tables' size is fixed by \( V \) and \( d \); the layers' size grows with depth. So the fraction falls as layers are added — at 120 layers the same tables would be about a third, and at 1200 about four percent. Scaling studies relate capability to parameter count and want a number that tracks computational depth; a count dominated by a vocabulary table would make two models with different tokenizers look like different sizes while computing identically. Hence non-embedding parameters.

The softmax of \( (a, 0) \) gives the first option \( \sigma(a) \), and \( \sigma'(a) = \sigma(a)(1-\sigma(a)) \).

a = 1 → p = 0.731, slope 0.197  ·  a = 4 → p = 0.982, slope 0.0177  ·  a = 10 → p = 0.99995, slope 0.0000454

A gap of 10 leaves the slope four thousand times smaller than a gap of 1. As the leading logit pulls away, the probability it produces stops responding to it, so the gradient reaching the logits vanishes and learning stalls — the network is confident and therefore deaf. Nothing has gone wrong with the parameters; the arithmetic of the exponential simply ran out of room. This is why every mechanism in the book that feeds a softmax also controls the scale of what it feeds it, from the \( 1/\sqrt{d} \) of Chapter 6 to the temperature of Section 6.4.

With every \( J_\ell = cI \), the product over \( L \) layers is \( c^L I \). At \( L = 50 \): \( 0.9^{50} = e^{50\ln 0.9} = e^{-5.27} = 5.2\times10^{-3} \), and \( 1.1^{50} = e^{50\ln 1.1} = e^{4.77} = 117 \).

A ten percent error in either direction, compounded fifty times, spans a factor of about \( 23{,}000 \) between the two outcomes — one gradient has all but vanished, the other has grown by two orders of magnitude. The usable range of \( c \) is therefore not a range at all but a knife edge near 1, and it narrows as \( L \) grows: holding \( c^L \) within a factor of ten of 1 requires \( c \) within about \( 2.3/L \) of unity, which at \( L = 50 \) is five percent and at \( L = 500 \) is half of one percent. Depth does not merely make this problem worse; it makes it worse in proportion to itself. Chapter 4 does not solve it by tuning \( c \) — it changes the factor from \( J_\ell \) to \( I + J_\ell \), so that the identity survives no matter what \( J_\ell \) does.

Take the expectation of the cross-entropy under the true distribution \( p \) and add and subtract \( \ln p_i \) inside the sum:

\( \mathbb{E}_p[-\ln q_i] = -\sum_i p_i \ln q_i = -\sum_i p_i \ln p_i + \sum_i p_i \ln\frac{p_i}{q_i} = H(p) + \mathrm{KL}(p\|q) \).

So the expected loss is the entropy of the data plus the divergence of the model from it. The second term is what training reduces, and it is non-negative and zero only when \( q = p \). The first term does not contain \( q \) at all: no model, of any size, trained on any amount of data, can push the loss below it.

In a language model \( H(p) \) is the entropy of language itself — the genuine unpredictability of the next token given everything before it, which no amount of modelling removes because it is a property of the text and not of the predictor. That floor is the \( E \) term in Chapter 7's loss surface \( L(N,D) = AN^{-a} + BD^{-b} + E \), and this is why it appears there as a constant rather than as something to be optimized away. It is also why it drops out of the compute-optimal derivation entirely: a term with no \( N \) and no \( D \) in it vanishes on differentiation, so the best split of a budget between model and data does not depend on how predictable language is.

(a) \( \eta < 2/\lambda_{\max} = 2/200 = 0.01 \). (b) At \( \eta = 0.009 \): along the steep axis the factor is \( |1 - 0.009(200)| = 0.8 \); along the shallow axis \( |1 - 0.009(2)| = 0.982 \). (c) The shallow direction needs \( \ln(0.01)/\ln(0.982) = 254 \) steps to contract a hundredfold. The steep direction reaches the same contraction in \( \ln(0.01)/\ln(0.8) = 21 \) steps — and then spends the remaining 233 steps oscillating about the valley floor, contributing nothing. That idle majority is what conditioning costs.

For large \( \kappa \), \( \rho_{\mathrm{GD}} = \frac{\kappa-1}{\kappa+1} \approx 1 - 2/\kappa \), so \( \ln(1/\rho_{\mathrm{GD}}) \approx 2/\kappa \). Similarly \( \rho_{\text{mom}} \approx 1 - 2/\sqrt\kappa \) gives \( \ln(1/\rho_{\text{mom}}) \approx 2/\sqrt\kappa \). Since step count is \( \ln F/\ln(1/\rho) \), the ratio of counts is \( \frac{2/\sqrt\kappa}{2/\kappa} = \sqrt\kappa \).

Per layer the variance multiplier is \( m\sigma^2 = 500 \times (1/1000) = 0.5 \). Over twelve layers the activation variance is multiplied by \( 0.5^{12} = 1/4096 = 2.4\times10^{-4} \) relative to the variance-preserving case. A single factor of \( \sqrt2 \) in \( \sigma \), compounded twelve times, costs nearly four orders of magnitude.

If the update along axis \( i \) becomes \( x_i \leftarrow (1-\eta)x_i \), every axis contracts at the same rate, so the effective condition number is 1 and one well-chosen step suffices in principle. Real second-order methods approximate the inverse curvature to achieve this; adaptive optimizers approximate it far more cheaply with per-coordinate scale estimates accumulated from gradient history. Both are answering the question this chapter posed: the step size should not be hostage to the single steepest direction.

Path length is \( \mathrm{Binomial}(100, \tfrac12) \): mean 50, variance 25, standard deviation 5. With continuity correction the endpoints standardize to \( (39.5-50)/5 = -2.1 \) and \( (60.5-50)/5 = +2.1 \). The enclosed fraction is \( 2\Phi(2.1) - 1 = 2(0.9821) - 1 = 0.964 \) — about 96% of paths traverse 40 to 60 blocks.

Plain: \( 18C^2 \). Bottleneck: \( C(C/c) + 9(C/c)^2 + (C/c)C = \frac{2C^2}{c} + \frac{9C^2}{c^2} \). Ratio \( = \frac{18}{2/c + 9/c^2} = \frac{18c^2}{2c + 9} \). At \( c = 2 \): \( 72/13 = 5.5 \). At \( c = 4 \): \( 288/17 = 16.9 \). For large \( c \) the ratio approaches \( 9c \), growing without bound — but so does the information lost in the narrow middle, which is why compression factors in practice stay modest.

Dilated: \( 2^{n+1} - 1 \ge 255 \Rightarrow n = 7 \) (giving exactly 255). Undilated: \( 2n+1 \ge 255 \Rightarrow n = 127 \). A single kernel of that field holds \( 255^2 = 65{,}025 \) weights per channel pair against the stack's \( 7\times9 = 63 \) — a ratio of about \( 1{,}030 \).

Accumulated variance \( 60 \times 0.02 = 1.2 \), so the standard deviation is \( \sqrt{1.2} = 1.10 \) — already larger than a unit-scale signal after sixty blocks.

Putting normalization on the skip path replaces the identity in Derivation 4.1 with a scaling: the \( k=0 \) term becomes \( \prod_\ell (1/\sqrt{\ell})\,I \) rather than \( I \), and that product decays without bound. The gradient floor, which was the point of the whole construction, is gone. So the drift argument does justify controlling the scale, and does not justify controlling it there — the branch is the only place where rescaling costs nothing, because the branch is the term the expansion can afford to lose.

Single \( 7\times7 \): \( 49 \times 256^2 = 3{,}211{,}264 \) weights and the same count of multiply–accumulates. Three \( 3\times3 \): \( 3 \times 9 \times 256^2 = 1{,}769{,}472 \) — the same \( 7\times7 \) receptive field for 55% of the cost, plus two extra nonlinearities. Bottleneck at \( c = 4 \): \( 256\cdot64 + 9\cdot64^2 + 64\cdot256 = 69{,}632 \), but its receptive field is only \( 3\times3 \).

Ranked by cost the bottleneck wins by a factor of 46, and the ranking is misleading: it is not doing the same job. Against equal receptive field the stack of three is the right comparison and it is both cheaper and more expressive. The bottleneck buys depth per unit cost rather than reach, which is why real networks use both — stacks of small kernels for reach, bottlenecks for depth.

In Derivation 4.1 the surviving \( k=0 \) term is the identity only if the skip path is genuinely unmodified; inserting any transformation there replaces \( I \) with that transformation's Jacobian, and a product of those can again vanish. Keeping the skip clean preserves the floor exactly. In Worked Example 4.2 the drift comes from the accumulated branch outputs; placing normalization inside the branch controls each contribution before it is added, whereas normalizing the skip would rescale the carefully preserved identity signal itself. The unifying principle: keep the pathway that guarantees signal flow free of anything that could attenuate it, and place all control on the pathway that adds new content.

(a) Recurrent-path dropout at keep-probability \( 0.9 \) over 200 steps leaves \( 0.9^{200} = e^{-21.1} = 7\times10^{-10} \) of the signal — total annihilation. (b) Vertical-only dropout perturbs a given piece of information once per layer, so three times, regardless of the sequence's length. The vertical design preserves memory; the difference is that one exposure count scales with time and the other with depth.

Require \( \ln(0.01)/\ln f = 500 \Rightarrow \ln f = -0.00921 \Rightarrow f = 0.9908 \). The bias is \( b = \ln\frac{0.9908}{0.0092} = \ln(107.7) = 4.68 \).

Embedding \( 1024 \times 30{,}000 = 30.72 \)M. Each recurrent layer \( 4 \times 1024 \times (1024+1024+1) = 4096 \times 2049 = 8.39 \)M, so three layers give \( 25.18 \)M. Output projection \( 1024\times30{,}000 + 30{,}000 = 30.75 \)M. Total \( \approx 86.7 \)M, of which embedding plus projection is \( 61.5 \)M — about \( 71\% \). Since that fraction scales with vocabulary rather than with computational depth, including it would corrupt any attempt to relate capability to model size; hence scaling studies count non-embedding parameters.

At \( f = 1 \), \( f^t = 1 \) for every \( t \): the horizon in Derivation 5.1 is unbounded, since \( \ln f = 0 \) makes the required \( t \) infinite. A cell fixed at \( f = 1 \) could never discard information, so its state would accumulate every input it ever saw and the useful signal would be swamped — and it could not represent tasks requiring the state to reset. Making \( f \) input-dependent lets the network hold near 1 when carrying information matters and drop it when the context changes, so the horizon becomes a per-token decision rather than an architectural constant.

Attention \( = 2n^2d = 2(8192)^2(1024) = 1.37\times10^{11} \). Feed-forward \( = 8nd^2 = 8(8192)(1024)^2 = 6.87\times10^{10} \). The ratio is \( 2:1 \) — attention dominates at this length. They balance at \( n = 4d = 4096 \), which is where the model crosses from feed-forward-dominated to attention-dominated.

Unreversed, source token \( j \) is read at step \( j \) and target token \( j \) emitted at step \( n+j \), so every lag is \( n \). Reversed, source token \( j \) is read at step \( n-j+1 \), so the lag is \( (n+j)-(n-j+1) = 2j-1 \). The mean is \( \frac{1}{n}\sum_{j=1}^{n}(2j-1) = \frac{n^2}{n} = n \) — unchanged. The maximum rises to \( 2n-1 \). It aids optimization because the earliest alignments become nearly immediate, giving gradient descent a foothold it can learn from at once; once the early alignments are established the rest follow. It does not aid representation, because the total distance information must travel is provably identical.

For \( T > 0 \), the map \( z \mapsto z/T \) is strictly increasing, so \( z_a > z_b \Rightarrow z_a/T > z_b/T \). The softmax is a strictly increasing function of its argument relative to the others (the exponential is increasing and the shared denominator is positive), so \( P(a) > P(b) \) for every positive \( T \). Temperature therefore alters only how concentrated the distribution is, never which candidate leads.

The principle: hold the variance of a signal at order one as it crosses any operation, because both vanishing and saturating regimes destroy gradients. In Chapter 3, the quantity is the pre-activation of a layer, held at unit variance by \( \sigma^2 = 1/m \); otherwise activations shrink or explode geometrically with depth. In Chapter 4, the quantity is the accumulated residual signal, whose \( \sqrt{L} \) drift is contained by normalizing inside the branch; otherwise the signal outgrows the scale the later layers were tuned for. In Chapter 6, the quantity is the attention logit, held at unit variance by dividing by \( \sqrt{d} \); otherwise the softmax saturates and its gradient vanishes. Same instinct, three operations.

Substituting \( D = C/(kN) \) gives \( L(N) = AN^{-a} + B(k/C)^b N^b \). Differentiating: \( -aAN^{-a-1} + bB(k/C)^bN^{b-1} = 0 \). Multiplying by \( N \) and rewriting the second term as \( bBD^{-b} \) yields \( aAN^{-a} = bBD^{-b} \). Combining with \( N^a \propto D^b \) and \( ND \propto C \): \( D^{1+b/a} \propto C \), so \( D \propto C^{a/(a+b)} \) and \( N \propto C^{b/(a+b)} \). The bare-term balance is wrong because the constraint's exchange rate between \( N \) and \( D \) depends on each term's responsiveness, which is exactly its exponent.

\( C = 6(7\times10^{10})(1.4\times10^{12}) = 5.88\times10^{23} \). At ratio 20, \( C = 120N^2 \Rightarrow N^2 = 4.9\times10^{21} \Rightarrow N = 7\times10^{10} \). The model is already at 70 B with \( D/N = 1.4\times10^{12}/7\times10^{10} = 20 \) exactly — it is compute-optimal as given, neither over- nor under-trained. (Contrast Worked Example 7.2, where the ratio was under 2 and the re-budgeting cut the model by a factor of three.)

Requiring \( k^{-a} = \tfrac12 \) gives \( k = 2^{1/a} \); at \( a = 0.076 \), \( k = 2^{13.16} \approx 9{,}100 \). Since compute scales with \( N \) (and, at fixed ratio, with \( N^2 \)), each halving of this loss term multiplies the training bill by four orders of magnitude or more. Steady progress on loss is therefore exponentially expensive in compute — the arithmetic behind the field's escalating training budgets.

\( D/N \propto C^{(a-b)/(a+b)} \). If \( a < b \) the exponent is negative and the optimal tokens-per-parameter ratio falls as budgets grow; if \( a > b \) it rises. Interpretation: the exponent measures how quickly a term responds to its resource, so a larger \( a \) means parameters buy improvement faster, and the optimum then shifts spending toward the slower-improving resource — data — to keep the weighted marginal returns equal. Real recipes have drifted toward far more tokens per parameter, but Chapter 17 shows the dominant reason is not this drift at all: it is the serving term entering the objective.

Micro-batch \( m \) enters stage 1 at time \( m-1 \) and advances one stage per unit, so it exits stage \( P \) at \( m + P - 1 \); the last exits at \( M + P - 1 \). Useful work is \( MP \) stage-units; capacity is \( P(M+P-1) \); the bubble is \( 1 - \frac{MP}{P(M+P-1)} = \frac{P-1}{M+P-1} \). At \( (8,1) \): \( 7/8 = 87.5\% \) idle — naive model parallelism wastes seven eighths of the fleet. At \( (8,32) \): \( 7/39 = 17.9\% \) — still nearly a fifth idle even with thirty-two micro-batches.

\( \frac{11}{M+11} \le 0.08 \Rightarrow M + 11 \ge 137.5 \Rightarrow M \ge 127 \).

\( 16 \times 1.75\times10^{11} = 2.8\times10^{12} \) bytes \( = 2{,}800 \) GB, requiring \( 2800/80 = 35 \) accelerators for the state alone. Computing the unsharded figure first is right because it is the quantity that decides whether the run is possible at all; the sharded per-device number tells you nothing about feasibility until you know how many devices the total demands.

At fixed \( M \), raising \( P \) increases the bubble \( \frac{P-1}{M+P-1} \) monotonically toward 1, while per-accelerator memory falls roughly as \( 1/P \). Pipelining therefore trades time for memory at a worsening rate. Real systems combine it with tensor parallelism (splitting individual operations, which costs communication rather than idle time) and with state sharding (which costs communication but no idle time at all), so that the memory requirement can be met without pushing \( P \) into the regime where the bubble dominates. The general lesson: when one axis of parallelism has a penalty that grows with its degree, use several axes at modest degree rather than one at high degree.

At \( k = 8 \): \( p = 0.5 \to 0.0039 \); \( 0.7 \to 0.058 \); \( 0.8 \to 0.168 \); \( 0.9 \to 0.430 \); \( 0.95 \to 0.663 \). The steepest rise is between \( 0.8 \) and \( 0.95 \), bracketing the 50% frontier \( 2^{-1/8} = 0.917 \). A per-token log-loss curve shows no cliff because it measures \( p \) itself, which moves smoothly; the cliff is created by raising that smooth quantity to the eighth power and comparing against a threshold.

\( 0.90 = c + 0.65(1-c) = 0.65 + 0.35c \Rightarrow c = 0.714 \). Then \( P(\text{contaminated}\mid\text{correct}) = 0.714/0.90 = 0.794 \). Nearly four fifths of the correct answers would be recall rather than skill — the headline score describes the test set's provenance more than the model.

Majority of three is correct with probability \( 3p^2(1-p) + p^3 = 3p^2 - 2p^3 \). This exceeds \( p \) iff \( 3p - 2p^2 > 1 \), i.e. \( 2p^2 - 3p + 1 < 0 \), i.e. \( (2p-1)(p-1) < 0 \). Since \( p < 1 \) makes the second factor negative, the product is negative exactly when \( 2p - 1 > 0 \), i.e. \( p > \tfrac12 \).

Emergence: the model owns \( p \); the ruler owns \( k \) and the exact-match threshold. Attributing the cliff to the model asserts a discontinuous capability change that the evidence does not support. Contamination: the model owns its clean accuracy; the ruler owns \( c \), a property of the test set's construction. Attributing the inflation to the model overstates skill by exactly \( c(1-p) \). Voting: the model owns \( p \); the procedure owns the number of samples and the aggregation rule. Attributing the gain to the model claims a capability increase where only variance was reduced. In all three, the observed number is a function of model and measurement jointly, and the error is always the same: crediting the model with the measurement's contribution.

P: \( 45 + 300 = 345 \). Q: \( 225 + 95 = 320 \). R: \( 450 + 0 = 450 \). MDL selects Q; the zero-error model R loses by 130 bits. Overfitting here is the quantity by which a model's parameter cost exceeds the residual bits those parameters save — a number, not a judgment.

Scale weights by \( 2^{14} \): the four programs carry \( 2^{8} = 256 \), \( 2^{5} = 32 \), \( 2^{5} = 32 \), and \( 1 \); total \( 321 \). Those predicting A hold \( 256 + 32 = 288 \), so \( P(A) = 288/321 = 0.897 \). General rule: a program \( \delta \) bits longer than another holds \( 2^{-\delta} \) times its weight. Proof: weights are \( 2^{-\ell} \), so the ratio of two is \( 2^{-\ell_1}/2^{-\ell_2} = 2^{\ell_2-\ell_1} \).

With \( n = 200 \): \( p = 100 \to 100/99 = 1.01 \); \( p = 150 \to 150/49 = 3.06 \); \( p = 190 \to 190/9 = 21.1 \); \( p = 195 \to 195/4 = 48.8 \). The risk accelerates violently as \( p \to n-1 \). The implication is that model sizes near the number of training samples are the worst possible choice — and that the classical instinct to stop growing the model before it interpolates lands you precisely in the danger zone.

(a) \( \ln(1/0.5) + \frac{0.25}{2} - 0.5 = 0.693 + 0.125 - 0.5 = 0.318 \) nats. (b) \( \ln(1/0.05) + \frac{0.0025}{2} - 0.5 = 3.00 + 0.001 - 0.5 = 2.50 \) nats. The precise weight costs nearly eight times as much. A description-length penalty therefore leaves imprecise every weight whose exactness does not purchase a commensurate reduction in residual bits — which is most of them.

Bits per character: \( 1.9/4.5 = 0.422 \); \( 1.5/3.2 = 0.469 \); \( 2.4/6.0 = 0.400 \). Ranked best to worst: the third, the first, the second. On \( 5\times10^9 \) characters: \( 264 \) MB, \( 293 \) MB, \( 250 \) MB respectively (bits divided by 8).

A per-token leaderboard ranks them 1.5, 1.9, 2.4 — exactly backwards. The best compressor has the worst per-token loss, because its tokens are the largest. Any comparison across tokenizers that does not divide by \( \gamma \) is measuring the vocabulary.

With \( \mu = 0 \) the cost is \( \ln(1/\sigma_q) + \sigma_q^2/2 - 1/2 \).

\( \sigma_q = 1 \to 0 \); \( 0.5 \to 0.693 + 0.125 - 0.5 = 0.318 \); \( 0.25 \to 1.386 + 0.031 - 0.5 = 0.918 \); \( 0.1 \to 2.303 + 0.005 - 0.5 = 1.808 \); \( 0.01 \to 4.605 + 0.00005 - 0.5 = 4.105 \) nats.

Once \( \sigma_q \) is small the \( \sigma_q^2/2 \) term is negligible and the cost is \( \ln(1/\sigma_q) - 1/2 \), so halving \( \sigma_q \) adds exactly \( \ln 2 \) in the limit. Check it on successive halvings: \( 0.5 \to 0.25 \) adds \( 0.600 \), \( 0.25 \to 0.125 \) adds \( 0.669 \), \( 0.125 \to 0.0625 \) adds \( 0.688 \) — converging on \( \ln 2 = 0.693 \) from below. Each further bit of precision therefore costs the same as the last rather than less, which is what makes precision worth buying only where it pays — there is no volume discount.

Model selection: model bits are the parameter encoding, residual bits the unexplained data. Hypothesis odds: model bits are the hypothesis specification, residual bits the negative log-likelihood. Weight precision: model bits are the coding cost of the posterior against the prior; residual bits are the loss the weight's imprecision causes. Cross-entropy as file size: only residual bits — the model is assumed already shared. The length prior: only model bits, since the programs considered fit perfectly. Typicality: the model is the constraint set, and residual bits are the deficiency between a typical member's length and this member's. Grokking: two competing model encodings at identical (zero) residual. Double descent: the parameter count is model bits and the noise-fitting is residual, with the divergence marking where residual savings stop justifying parameters. A single principle wears this many disguises because any inference problem can be posed as choosing a description, and the only currency a description has is its length. The reading lesson: when a new result seems novel, ask what it is trading between model bits and residual bits — often the novelty is in the encoding, not the principle.

(a) Weights: \( 7\times10^{10}\times2 = 140 \) GB. (b) Cache: \( 2\times80\times8\times128\times8192\times16\times2 = 4.3\times10^{10} \) bytes \( = 43 \) GB. (c) Sum \( = 183 \) GB. (d) At 80 GB each, three accelerators — barely, with no room for activations, so four in practice. Attack the weights first: quantization to 8 bits halves the dominant term and simultaneously doubles the decode ceiling, whereas cache reductions help only at long context or large batch.

Batch-\( B \) decode performs about \( 2NB \) operations while still moving about \( 2N \) bytes of weights (read once, reused across the batch), so intensity \( \approx B \). Compute-bound requires \( B \gtrsim 156 \) on the reference machine. As batch size approaches that, throughput rises nearly linearly while per-request latency degrades, because each request waits for the whole batch — the fundamental throughput-versus-latency tension in serving.

140 / 70 / 35 GB at 16 / 8 / 4 bits, giving ceilings of \( 2000/140 = 14.3 \), \( 28.6 \), and \( 57 \) tokens per second. Quantization improves the bytes-moved budget and leaves the operation count essentially unchanged. Decode is bandwidth-bound, so shrinking bytes raises its ceiling proportionally; prefill is compute-bound, so shrinking bytes does not help it. Same change, opposite consequences, decided entirely by which side of the ridge the workload sits on.

Take a model trained compute-optimally at, say, 70 B parameters, versus a smaller 35 B model trained on proportionally more data to equal quality. Chapter 7's objective is indifferent to serving and picks whichever minimizes training cost. But by this chapter's formulas the 35 B model has half the weight memory, twice the decode ceiling, and a proportionally smaller cache — advantages paid on every request forever. The missing term is the serving cost: the objective must become \( 6ND_{\text{train}} + 2NT \) for a lifetime of \( T \) generated tokens, which is exactly Chapter 17's construction.

Score-matrix traffic (written once, read once) is \( 2\times 2n^2 = 4n^2 \) bytes \( = 4(4096)^2 = 6.7\times10^7 \). The unavoidable traffic for queries, keys, values, and output is \( 4\times2nd = 8(4096)(128) = 4.2\times10^6 \). The ratio is \( \frac{4n^2}{8nd} = \frac{n}{2d} = 16:1 \). It grows linearly in \( n \) and falls inversely with \( d \), so the technique matters most at long context and modest head dimension — exactly the regime that became standard.

With \( p = (0.3, 0.7) \), \( q = (0.5, 0.5) \): acceptance for symbol 1 is \( \min(1, 0.6) = 0.6 \); for symbol 2, \( \min(1, 1.4) = 1 \). Accepted mass \( = 0.5(0.6) + 0.5(1) = 0.8 \), so rejection occurs with probability \( 0.2 \). The residual \( \max(0, p-q) = (0, 0.2) \) normalizes to \( (0, 1) \). Emitted probabilities: symbol 1 gets \( 0.5(0.6) = 0.3 \) ✓; symbol 2 gets \( 0.5(1) + 0.2(1) = 0.7 \) ✓. Exactly \( p \).

With \( \alpha = 0.8 \), \( c = 0.1 \): \( k=2 \) gives \( 2.44/1.2 = 2.03\times \); \( k=4 \) gives \( 3.36/1.4 = 2.40\times \); \( k=8 \) gives \( 4.33/1.8 = 2.41\times \); \( k=16 \) gives \( 4.89/2.6 = 1.88\times \). The optimum is a broad plateau around \( k \approx 6 \) to \( 9 \). The marginal condition \( \alpha^{k+1} \approx c \) gives \( 0.8^{k+1} = 0.1 \Rightarrow k+1 = 10.3 \Rightarrow k \approx 9 \), consistent with the plateau's upper end.

Prefill already processes many positions in one pass, so its arithmetic intensity is high and it sits above the ridge point — the accelerator is already busy, and there is no idle capacity for speculation to exploit. Decode sits far below the ridge, so verifying \( k \) positions at once costs almost the same as verifying one, and the speculation converts wasted capacity into tokens. The general condition: speculation helps a workload whose per-step arithmetic intensity is far below the ridge point and whose steps are sequentially dependent, so that batching within a single request is otherwise impossible. Autoregressive sampling from any large model qualifies; so does sequential simulation where a cheap surrogate can propose several steps for a costly verifier to check in parallel.

Shared components: \( 0.2\times140 = 28 \) B. Expert parameters: \( 112 \) B across 64 experts \( = 1.75 \) B each; top-2 routing activates \( 3.5 \) B. Active total \( \approx 31.5 \) B. Behaves like the active figure: training operations and decode arithmetic. Behaves like the total figure: serving memory, since every expert must be resident. The KV cache belongs to neither — it depends on layers, key/value heads, head dimension, and context, and is unaffected by expert count.

MSE scales as \( R^2 \), so the range moving from 1 to 6 multiplies the error by 36. Each additional bit quarters the error, so restoring it needs \( 4^b \ge 36 \Rightarrow b = \log_4 36 = 2.58 \), i.e. 3 more bits everywhere. Handling outliers separately is cheaper because the extra precision is then spent only on the few channels that need it, rather than on every weight in the matrix — the same allocation logic as Chapter 10's account of paying for precision only where it buys accuracy.

(a) Weights: \( 1.3\times10^{10}\times2 = 26 \) GB. (b) Full fine-tuning optimizer state: \( 16\times1.3\times10^{10} = 208 \) GB. (c) Rank-16 adaptation of matrices totalling 30% of the model makes well under one percent of parameters trainable, so optimizer state falls to roughly 1–2 GB. To fit training on a single 80 GB accelerator, the binding constraint is (b), so low-rank adaptation is the necessary technique; quantizing the frozen base then reduces (a) as well, and the two compose.

With \( E = 2 \) and \( f_1 = P_1 = x \), \( f_2 = P_2 = 1-x \): \( \mathcal{L} = 2[x^2 + (1-x)^2] \). Differentiating, \( 4x - 4(1-x) = 0 \Rightarrow x = \tfrac12 \), giving \( \mathcal{L} = 2(\tfrac14+\tfrac14) = 1 \). The second derivative is positive, so this is the minimum. Why both mechanisms: the auxiliary loss is a soft gradient signal that shapes the router over training but guarantees nothing at any particular step, so alone it would permit transient severe imbalance — and early in training, when the router is near-random, that imbalance can be self-reinforcing as favored experts improve fastest. The capacity limit alone would prevent overload but supplies no gradient pressure toward balance, so the router would keep proposing imbalanced assignments and the system would silently drop tokens forever. Pressure plus a hard limit gives both a direction and a bound.

With \( \epsilon = 0.2 \), clip range \( [0.8, 1.2] \).

\( A = +1 \): at \( \rho = 0.7 \), \( \min(0.7, 0.8) = 0.7 \), unclipped branch, gradient alive. At \( \rho = 1.0 \), both give 1.0, alive. At \( \rho = 1.3 \), \( \min(1.3, 1.2) = 1.2 \), clipped, gradient dead.

\( A = -1 \): at \( \rho = 0.7 \), \( \min(-0.7, -0.8) = -0.8 \), clipped, gradient dead. At \( \rho = 1.0 \), both \( -1.0 \), alive. At \( \rho = 1.3 \), \( \min(-1.3, -1.2) = -1.3 \), unclipped, alive.

The pattern: the gradient dies exactly when the policy has already moved beyond the trust region in the direction the advantage favors. A single update can therefore never be rewarded for moving further than \( \epsilon \) in the profitable direction, which bounds how far one batch can push the policy.

The optimum is \( \pi^*(y\mid x) = \pi_{\text{ref}}(y\mid x)e^{r(x,y)/\beta}/Z(x) \). Inverting: \( r(x,y) = \beta\log\frac{\pi^*(y\mid x)}{\pi_{\text{ref}}(y\mid x)} + \beta\log Z(x) \). Substituting into \( P(y_w \succ y_l) = \sigma(r(x,y_w) - r(x,y_l)) \), the two \( \beta\log Z(x) \) terms appear with opposite signs and cancel, because \( Z \) depends on \( x \) alone and both responses share the prompt. The loss is \( -\log\sigma\bigl(\beta\log\frac{\pi(y_w)}{\pi_{\text{ref}}(y_w)} - \beta\log\frac{\pi(y_l)}{\pi_{\text{ref}}(y_l)}\bigr) \). If the two responses came from different prompts, the normalizers would be \( Z(x_1) \) and \( Z(x_2) \), which do not cancel — and the method would be intractable, since \( Z \) sums over all possible responses.

Policy alone: \( 20 \times 7\times10^9 = 1.4\times10^{11} \) bytes \( = 140 \) GB. Policy plus a same-sized critic: \( 280 \) GB. The ratio is 2 — the critic is half the total bill, which is precisely the resource that group-relative advantage estimation recovers by computing the baseline from sampled rewards instead of a learned network.

If every \( r_i = r \), then \( \bar r = r \) and each numerator \( r_i - \bar r = 0 \), so every advantage is zero regardless of \( r \) (the standard deviation is also zero, but the numerator vanishing is what matters — the update is identically zero). Consequence for curriculum: prompts the model always solves and prompts it never solves both produce all-equal groups and contribute nothing. Training must therefore be concentrated on problems near the model's current success boundary, and the boundary moves as the model improves — so the curriculum must be continuously re-selected rather than fixed in advance.

Both are instances of: a quantity that enters an expression only through a difference of two terms in which it takes the same value is unidentifiable from, and irrelevant to, that expression. In Section 14.2 the additive reward constant is shared by both responses and cancels in \( r_w - r_l \), so no data can determine it. In Section 14.4 the log-normalizer is shared by both responses to the same prompt and cancels in the same difference, so its intractability does not matter. A third instance in this book: Chapter 7's irreducible loss term \( E \), which is constant in both \( N \) and \( D \) and therefore vanishes on differentiation, so the compute-optimal allocation is entirely independent of it. A fourth: Chapter 10's normalization of posterior weights, where a common scaling factor across all programs cancels in the ratio — which is exactly why we may rescale by \( 2^{\ell_{\max}} \) and work in integers.

Budget 24: the small model gets 24 samples, coverage \( 1 - 0.85^{24} = 1 - e^{-3.90} = 0.980 \). The large model's single attempt exhausts the same budget and delivers \( 0.55 \), with nothing to select between. Delivered accuracy for sampling: \( v = 1.0 \to 0.980 \); \( v = 0.8 \to 0.784 \); \( v = 0.6 \to 0.588 \). They tie when \( 0.980v = 0.55 \), i.e. \( v = 0.561 \). Above that selector reliability, sampling the small model wins; below it, the large model's single answer is better.

Majority of three: \( 3p^2(1-p) + p^3 = 3p^2 - 2p^3 \). Exceeds \( p \) iff \( 3p - 2p^2 > 1 \) iff \( (2p-1)(p-1) < 0 \) iff \( p > \tfrac12 \). At \( p = 0.45 \): \( 3(0.2025) - 2(0.0911) = 0.425 < 0.45 \) — voting made it worse. At \( p = 0.55 \): \( 3(0.3025) - 2(0.1664) = 0.575 > 0.55 \) — better. The threshold is exact, and below it the procedure reliably amplifies error.

The \( (k+1) \)-th sample converts a failure into a success with probability \( (1-p)^k p \), worth \( V(1-p)^kp \), against a cost \( c \). Continue while \( V p (1-p)^k > c \). With \( p = 0.1 \), \( V = 100 \), \( c = 1 \): \( 10(0.9)^k > 1 \Rightarrow 0.9^k > 0.1 \Rightarrow k < \ln(0.1)/\ln(0.9) = 21.9 \). So draw about 22 samples; beyond that the marginal sample costs more than the expected value it adds.

At temperature approaching zero the sampler becomes deterministic: every sample is the argmax, so all \( k \) samples are identical. Coverage collapses from \( 1-(1-p)^k \) to simply \( p \), because the samples are perfectly correlated rather than independent — the formula's premise has failed. Temperature therefore controls the diversity of samples, and diversity is precisely the resource that inference-time scaling consumes: without it there is nothing for a selector to select among. Connecting to Chapter 6: temperature is a monotone rescaling of the logits, so it cannot change which candidate is most probable — it changes only how much probability mass sits away from that leader. That mass is exactly what makes repeated sampling explore, which is why the quantity temperature controls is the one that matters here, and why very low temperatures and inference-time scaling are fundamentally incompatible.

\( \alpha_1 = 0.99 \), \( \alpha_2 = 0.97 \), so \( \bar\alpha_2 = 0.9603 \). The signal coefficient is \( \sqrt{0.9603} = 0.980 \) and the noise coefficient \( \sqrt{1-0.9603} = \sqrt{0.0397} = 0.199 \); the squared coefficients sum to \( 0.9603 + 0.0397 = 1 \) ✓. The induction step: assuming \( x_{t-1} = \sqrt{\bar\alpha_{t-1}}x_0 + \sqrt{1-\bar\alpha_{t-1}}\,\epsilon \), substituting into \( x_t = \sqrt{\alpha_t}x_{t-1} + \sqrt{1-\alpha_t}\,\epsilon_t \) gives signal coefficient \( \sqrt{\alpha_t\bar\alpha_{t-1}} = \sqrt{\bar\alpha_t} \) and total noise variance \( \alpha_t(1-\bar\alpha_{t-1}) + (1-\alpha_t) = 1 - \bar\alpha_t \).

Require \( (1-\beta)^{200} = 0.5 \Rightarrow 1-\beta = 0.5^{1/200} = e^{-0.003466} = 0.99654 \Rightarrow \beta = 0.00346 \). At \( t = 500 \): \( \bar\alpha = 0.99654^{500} = e^{-1.733} = 0.177 \), so the signal-to-noise ratio is \( 0.177/0.823 = 0.215 \).

Guidance requires both a conditional and an unconditional prediction at each step, so it doubles the per-step cost. With a stride of 20 on a 1000-step schedule there are 50 steps, hence \( 50 \times 2 = 100 \) network evaluations. Unguided, unstrided sampling would take 1000. So guided strided sampling is ten times cheaper than the naive baseline despite paying a factor of two for guidance — the stride is doing all the work and more.

Bottleneck block: channels are compressed by \( 1\times1 \) convolutions; the expensive spatial convolution runs in the narrow space; the budget saved is arithmetic; compressing too far starves the block's representational capacity. Expert routing: the token population is compressed — each token uses only \( k \) of \( E \) experts; the router does the compressing and the residual stream carries information around it; the budget saved is per-token compute; routing too sparsely starves experts of training signal and invites dropped tokens. Latent diffusion: the image is compressed by an encoder; the entire denoising process runs in the latent; the decoder restores resolution; the budgets saved are compute and activation memory; compressing too aggressively discards detail the decoder cannot invent. The principle: perform the expensive, repeated computation in the smallest representation that still carries what that computation needs, and let cheap machinery move between that representation and the full one.

P: \( (ND) = 2\times10^{23} \), so training \( = 1.2\times10^{24} \). Q: \( (ND) = 2.4\times10^{23} \), training \( = 1.44\times10^{24} \).

At \( T = 10^{12} \): \( C_P = 1.2\times10^{24} + 2\times10^{23} = 1.4\times10^{24} \); \( C_Q = 1.44\times10^{24} + 8\times10^{22} = 1.52\times10^{24} \). P wins.

At \( T = 10^{14} \): \( C_P = 1.2\times10^{24} + 2\times10^{25} = 2.12\times10^{25} \); \( C_Q = 1.44\times10^{24} + 8\times10^{24} = 9.44\times10^{24} \). Q wins.

Break-even: \( T^* = \frac{6(2\times10^{23} - 2.4\times10^{23})}{2(4\times10^{10} - 10^{11})} = \frac{-2.4\times10^{23}}{-1.2\times10^{11}} = 2\times10^{12} \) tokens. Recommend Q if you expect to serve more than about two trillion tokens over the model's life — a threshold most production deployments cross quickly, so Q unless the model is experimental or narrowly used.

At \( T = 5\times10^{12} \): (80 B, 1 T) gives \( 4.8\times10^{23} + 8\times10^{23} = 1.28\times10^{24} \). (40 B, 3 T) gives \( 7.2\times10^{23} + 4\times10^{23} = 1.12\times10^{24} \). (20 B, 10 T) gives \( 1.2\times10^{24} + 2\times10^{23} = 1.4\times10^{24} \). (10 B, 40 T) gives \( 2.4\times10^{24} + 10^{23} = 2.5\times10^{24} \). Best is (40 B, 3 T). The large extreme loses because serving a big model over a long life is expensive; the small extreme loses because matching quality with very few parameters demands so much data that training cost explodes. The optimum is interior because one term grows and the other shrinks as size falls.

Three further advantages of the 35 B option: its weight memory is halved (70 GB against 140 GB at half precision), which may change how many accelerators a replica needs; its batch-1 decode ceiling doubles, since the ceiling is bandwidth over weight bytes — roughly 29 tokens per second against 14 on the reference machine; and its KV cache per token is smaller in proportion to its layer count, easing long-context serving. An operations-only account understates the case because it prices only arithmetic, while the binding constraint in decoding is bandwidth and memory — the very budgets that shrink fastest as the model does.

A workable form: minimize \( \lambda_{\text{compute}}\bigl[6ND + 2NT\bigr] + \lambda_{\text{human}}\,H \) subject to a quality target \( L(N, D, H) \le L^\star \), where \( H \) is the quantity of preference data and \( \lambda \) are per-unit prices. The empirical inputs this book has taken as given: the loss surface's coefficients and exponents \( A, a, B, b, E \); the shape of the iso-quality curve that lets one trade \( N \) against \( D \); and, hardest of all, how quality depends on \( H \) — the return to additional human preference data. That last is hardest because preference data is heterogeneous in quality, its value depends on where it sits relative to the model's current failures, and it interacts with the other two variables rather than contributing independently. This is exactly the boundary at which derivation must hand off to measurement, and recognizing that boundary honestly — rather than extrapolating a fitted curve past its evidence — is the most valuable judgment the subject asks for.