← The Arithmetic of Intelligence
← Chapter 17Appendix B →
Appendix A

The Derivation Bank

The syllabus of this book, in 50 items. You are ready when each comes off a blank page, unprompted, with given constants slotted in.

This is not a summary to read. It is a checklist to be tested against — the statement of each result, without its derivation, so that you must supply the derivation yourself.

Use it three ways. As a gate: at the end of each chapter, cover the book and reproduce that chapter's entries. As a sweep: at the end of each Part, reproduce every entry so far in a few sittings. As a diagnostic: when a new paper resists you, scan this list and ask which entry prices it. Three items — marked protocol — are to be reproduced from nothing on three separate days, without rereading between attempts.

Part I · Foundations

  1. 1Training and serving cost: 2 FLOPs per multiply–accumulate, one forward pass and two backward, giving \( 6ND \) to train and \( 2N \) per generated token to serve. Ch. 2

Part II · The Classical Era

  1. 2The stable step-size window \( 0 < \eta < 2/\lambda_{\max} \), derived from \( |1-\eta\lambda| < 1 \), and why the steepest curvature binds. Ch. 3
  2. 3Steps to converge, \( t = \ln F/\ln(1/\rho) \), with the given rates \( \rho_{\mathrm{GD}} = \frac{\kappa-1}{\kappa+1} \) and \( \rho_{\text{mom}} = \frac{\sqrt\kappa-1}{\sqrt\kappa+1} \); the speedup ratio tends to \( \sqrt\kappa \). protocol Ch. 3
  3. 4Fan-in variance \( m\sigma^2 \) and the choice \( \sigma^2 = 1/m \); the symmetry-breaking argument for random initialization. Ch. 3
  4. 5Convolution arithmetic — output size with the floor, parameters with the bias, multiply–accumulates — chained through a network. Ch. 4
  5. 6Expansion of \( \prod_\ell (I + J_\ell) \): \( \binom{L}{k} \) terms with \( k \) Jacobians, and the indestructible \( k=0 \) identity. Ch. 4
  6. 7Residual path lengths as \( \mathrm{Binomial}(L,\tfrac12) \), with the normal approximation and continuity correction. Ch. 4
  7. 8The \( \sqrt{L} \) activation drift from independent per-block perturbations, and where normalization must sit. Ch. 4
  8. 9Bottleneck-versus-plain multiply–accumulate ratio at general channel compression \( c \). Ch. 4
  9. 10Dilated receptive field: a dilation-\( r \) layer adds \( 2r \), so the schedule \( 1,2,\dots,2^{n-1} \) reaches \( 2^{n+1}-1 \) against \( 2n+1 \) undilated; and the parameter-ratio payoff. Ch. 4
  10. 11Gradient horizon \( t > \ln\epsilon/\ln f \); the gate bias \( b = \ln\frac{f}{1-f} \); dropout compounding per time step versus per layer. Ch. 5
  11. 12Recurrent layer parameter count \( 4d_h(d_{\text{in}} + d_h + 1) \), assembled into a full model, and the embedding-plus-readout fraction. Ch. 5
  12. 13Power-law slope extraction from a “per tenfold” statement, and extrapolation to a target improvement. Ch. 5
  13. 14Score variance \( \operatorname{Var}(q\cdot k) = d \) forcing the \( 1/\sqrt{d} \) scale; attention \( 2n^2d \) and feed-forward \( 8nd^2 \) costs and their crossover at \( n = 4d \); temperature as monotone logit scaling. Ch. 6
  14. 15Reversal-trick lags: unreversed lag \( n \) everywhere, reversed lag \( 2j-1 \), mean exactly \( n \), maximum worse. Ch. 6
  15. 16The transformer block's parameter count: \( 3d^2 \) of query, key and value projections with the head count cancelling, \( d^2 \) of output projection and \( 8d^2 \) of feed-forward — \( 12d^2 \) per layer. Ch. 6
  16. 17The compute-optimal allocation: substitute, differentiate, obtain \( aAN^{-a} = bBD^{-b} \), and read off \( N \propto C^{b/(a+b)} \), \( D \propto C^{a/(a+b)} \). protocol Ch. 7
  17. 18The halving cost \( 2^{1/\alpha} \) for a loss term \( \propto N^{-\alpha} \), and re-budgeting a model at a target token ratio under fixed compute. Ch. 7
  18. 19Pipeline completion time \( M + P - 1 \) from a timing diagram, and the bubble fraction \( \frac{P-1}{M+P-1} \) with both boundary cases. Ch. 8
  19. 20Optimizer-state sharding arithmetic, always with the unsharded total computed first. Ch. 8
  20. 21The sixteen bytes per parameter of mixed-precision training, itemised, and which of them freezing a parameter removes. Ch. 8
  21. 22Activation memory per token per layer, and the checkpointing optimum \( g = \sqrt{L} \) from minimising \( g + L/g \), bought with about a third more arithmetic. Ch. 8
  22. 23Communication per axis of parallelism: \( 2N \) bytes per device per step for data parallelism, activations per layer for tensor parallelism, stage boundaries for pipelining. Ch. 8
  23. 24Exact-match accuracy \( p^k \), its 50% frontier at \( p = 2^{-1/k} \), and the manufactured-cliff argument. Ch. 9
  24. 25Contamination: observed \( = c + p(1-c) \), inverted for \( c \), and the Bayes flip \( P(\text{contaminated}\mid\text{correct}) = c/\text{observed} \). Ch. 9
  25. 26Two-part codes: model bits plus residual bits, model selection by the minimum, and hypothesis odds \( 2^{\Delta} \). Ch. 10
  26. 27Cross-entropy as a literal file size; tokens times bits per token divided by eight. Ch. 10
  27. 28The Gaussian coding cost of a weight, its zero case, its divergence as precision rises, and the noisy-weights-are-cheap reading. Ch. 10
  28. 29The length prior \( 2^{-\ell} \): Kraft, posterior mixtures over consistent programs, and the theorem that each extra bit halves the vote. Ch. 10
  29. 30Typicality versus membership, via randomness deficiency \( \log_2|M| - K(x) \). Ch. 10
  30. 31The OLS excess risk \( \sigma^2 p/(n-p-1) \) and its divergence at the interpolation threshold; grokking as a code-length transition. Ch. 10
  31. 32The universal intelligence measure \( \Upsilon = \sum_i 2^{-K_i}V_i \) with its three properties. Ch. 10
  32. 33The reward-seizing threshold \( r > (1-\gamma)/\gamma \), from the geometric series. Ch. 10
  33. 34Bits per token divided by the tokenizer's compression ratio gives bits per character, the only figure that compares two models with different vocabularies. Ch. 10

Part III · The Modern Era

  1. 35KV cache size \( 2\,L\,H_{kv}\,d_{\text{head}}\,s\,B\,\beta \), derived from what is stored and per what. Ch. 11
  2. 36Arithmetic intensity and the ridge point; the intensity of a large matmul, of batch-1 decode, and of batch-\( B \) decode. Ch. 11
  3. 37The decode ceiling — bandwidth divided by weight bytes — and the model FLOP utilization formula. Ch. 11
  4. 38Streaming softmax: the running maximum and normalizer update, with the value accumulator's rescaling, and a numerical verification that it is exact. Ch. 12
  5. 39Speculative decoding's exactness: acceptance at \( \min(1, p/q) \), residual resampling, and the proof that the emitted distribution is \( p \). Ch. 12
  6. 40Expected tokens per verification cycle \( \frac{1-\alpha^{k+1}}{1-\alpha} \), the speedup expression, and the marginal condition for \( k \). Ch. 12
  7. 41Mixture-of-experts accounting: total versus active parameters, the auxiliary balance loss and its uniform minimum, and capacity-based dropping. Ch. 13
  8. 42Quantization error \( \Delta^2/12 \), with the quarter-per-bit law and the \( R^2 \) range sensitivity. Ch. 13
  9. 43Low-rank adaptation's \( 2rd \) trainable parameters, and the three-budget map of which technique relieves which resource. Ch. 13
  10. 44Preference modelling: the logistic comparison loss, identifiability up to a constant, and the KL-tethered objective. Plus the clipped surrogate's four regimes and where the gradient dies. Ch. 14
  11. 45The preference-optimization collapse: closed-form optimum, inversion for the reward, substitution into the comparison model, and the exact cancellation of the normalizer. Plus group-relative advantages and the degenerate-group case. protocol Ch. 14
  12. 46Coverage \( 1-(1-p)^k \) with a selector's multiplier, and the voting threshold at \( p = \tfrac12 \) below which majority vote amplifies error. Ch. 15
  13. 47Eliciting versus creating: which interventions redistribute probability the model already has, and which change \( p \) itself. Ch. 15
  14. 48The diffusion forward process \( q(x_t\mid x_0) \) by Gaussian composition, the signal-to-noise schedule, and the variance check that the coefficients square to one. Ch. 16
  15. 49Guidance as extrapolation beyond the conditional prediction, and the stride cost \( \lceil T/k \rceil \). Ch. 16
  16. 50The lifecycle cost \( 6ND + 2NT \), its break-even lifetime, and the interior optimum that makes "train smaller and longer" a calculation rather than a slogan. Ch. 17

Synthesis

Four essays, to be written once, at the end. Each is a page, and each is a test of whether the bank has become a way of seeing rather than a list.

  1. S1Restate a modern quantization method's goal in the language of entry 23 — precision purchased where precision pays.
  2. S2Explain the mixture-of-experts auxiliary loss as a pressure toward routing entropy, and say what the capacity limit adds that the loss alone cannot provide.
  3. S3Diagram a modern reasoning-model training pipeline as a dataflow, labelling every arrow with the entry above that prices it.
  4. S4The four budgets — training operations, serving operations, memory bytes and bandwidth, human preference information. For each, name which chapters price it and which techniques arbitrage it against the others.