Eight small reflexes carry the entire book. None is difficult; all must become automatic before the machine learning begins.
Eight small pieces of mathematics carry this entire book. You almost certainly know all of them already. What you may not have is the ability to use them without thinking — and that, not knowledge, is what the chapters ahead will demand.
A physicist solving a mechanics problem does not rederive the chain rule; it is furniture in the mind, and attention goes to the physics. The same must be true here. When Chapter 7 asks you to minimize a loss under a compute constraint, the minimization itself cannot be where you spend effort — the insight is in what the answer means. So we drill the furniture first. Read each section, work its drills until they are mechanical, and pass the gate at the end before opening Chapter 2. If the gate is trivial for you, skip this chapter entirely; it exists only to remove obstacles, not to teach anything you will be tested on for its own sake.
The eight reflexes are: manipulating logarithms and power laws; the algebra of variance; the binomial distribution and its normal approximation; constrained minimization by substitution; geometric series; counting in bits; turning a conditional around; and counting operations. Each recurs a dozen times in what follows, and I flag the first major appearance of each as we go.
Nearly everything about scale — how loss falls as models grow, how error falls as data grows, how a probability compounds over a thousand steps — is a statement about power laws, and every power law is a straight line once you take logarithms. The reflex to build is moving between the two views without hesitation.
The one manipulation you will perform most often is solving \( a^t = b \) for the exponent. Take logarithms of both sides: \( t \ln a = \ln b \), so \( t = \ln b / \ln a \). The base does not matter as long as it is the same on top and bottom. A special case worth having in your fingers: the number of times you must multiply by a factor \( f<1 \) to fall below a threshold \( \epsilon \) is \( t = \ln \epsilon / \ln f \), and because both logarithms are negative the ratio is positive.
The second manipulation is extracting an exponent from a verbal description of a power law. Suppose a quantity \( y \) depends on \( x \) as \( y \propto x^{\beta} \), and you are told that multiplying \( x \) by ten multiplies \( y \) by \( 0.6 \) — a “forty percent relative reduction per tenfold increase.” Then \( 10^{\beta} = 0.6 \), so \( \beta = \log_{10} 0.6 \approx -0.22 \). Once you have \( \beta \), any other question about the law — how much \( x \) must grow to halve \( y \), say — is one more line of the same algebra. This exact move recovers a speech-recognition data law in Chapter 5 and the compute-optimal exponents in Chapter 7.
A probability that compounds
A recurrent network multiplies a gradient by \( 0.95 \) at each time step. After how many steps has the gradient shrunk below one percent of its initial size?
We need the smallest \( t \) with \( 0.95^{t} < 0.01 \). Taking logarithms, \( t > \ln(0.01)/\ln(0.95) = (-4.605)/(-0.0513) = 89.8 \), so \( t = 90 \). A factor that looks almost harmless — ninety-five hundredths — erases the signal in about ninety steps. Chapter 5 makes this the whole reason the LSTM was invented.
Three constants are worth committing to memory so these estimates need no calculator: \( \ln 2 = 0.693 \), \( \log_{10} 2 = 0.301 \), and \( \ln 10 = 2.303 \). From the first two you can reconstruct most of what you need; for instance, the number of bits in a quantity is its natural-log information divided by \( 0.693 \).
(a) Solve \( 0.9^{t} = 0.5 \). (b) A loss term behaves as \( L \propto N^{-0.076} \); by what factor must \( N \) grow to cut this term by one quarter? (c) A model reaches a perplexity of 64; what is its cross-entropy in bits per token? (Perplexity is \( 2^{H} \) with \( H \) in bits.)
(a) \( t = \ln 0.5 / \ln 0.9 = (-0.693)/(-0.105) = 6.58 \). (b) We need \( N^{-0.076} \) to reach \( 0.75 \) of its value, i.e. \( k^{-0.076} = 0.75 \) where \( N \) grows by factor \( k \); so \( k = 0.75^{-1/0.076} = 0.75^{-13.16} \). Since \( \ln 0.75 = -0.288 \), \( \ln k = 0.288 \times 13.16 = 3.79 \), \( k = e^{3.79} \approx 44 \). (c) \( H = \log_2 64 = 6 \) bits per token.
When many small independent effects combine, their variances add — not their standard deviations. This single fact, applied without fear, explains why a signal grows like the square root of depth, why attention scores must be rescaled by the square root of dimension, and why noise injected at every step of a recurrence is so much more destructive than noise injected once per layer. It is the most quietly powerful tool in the book.
Three rules suffice. First, for independent random variables the variance of a sum is the sum of variances: \( \operatorname{Var}(\sum_i X_i) = \sum_i \operatorname{Var}(X_i) \). Second, scaling a variable scales its variance by the square: \( \operatorname{Var}(aX) = a^2 \operatorname{Var}(X) \). Third, the product of two independent variables, each with zero mean and unit variance, itself has unit variance. From these three, every result we need follows in a line or two.
The variance of a weighted sum of inputs
Consider a single artificial neuron that forms the weighted sum \( s = \sum_{i=1}^{m} w_i x_i \) of \( m \) inputs. Suppose the inputs are independent with unit variance, and the weights are independent, drawn with mean zero and variance \( \sigma^2 \). What is the variance of \( s \)?
Each term \( w_i x_i \) is a product of independent zero-mean variables, so it has mean zero and variance \( \operatorname{Var}(w_i)\operatorname{Var}(x_i) = \sigma^2 \cdot 1 = \sigma^2 \). The terms are independent of one another, so their variances add:
\[ \operatorname{Var}(s) = \sum_{i=1}^{m} \sigma^2 = m\sigma^2. \]This is the entire content of “fan-in scaling.” If we want the neuron's output to have the same scale as its inputs — neither exploding nor collapsing as signal passes through the layer — we must choose \( \sigma^2 = 1/m \). We will meet this exact condition again in Chapter 3 as the principle behind sensible initialization, and its cousin in Chapter 6, where the fan-in is a vector dimension and the same square root sets the scale of attention.
Two consequences deserve to be stated on their own, because each is a derivation you will be asked to reproduce. The dot product of two random \( d \)-dimensional vectors, each component independent with unit variance, has variance \( d \) — just apply the derivation above with \( m = d \) and unit weights. And the accumulation of \( L \) independent perturbations, each of variance \( v \), has variance \( Lv \) and therefore standard deviation \( \sqrt{Lv} \): a drift that grows like the square root of the number of contributions. Chapter 4 uses precisely this to explain why signals swell as they pass down a hundred-layer residual stack.
(a) Two random vectors of dimension 288 have independent unit-variance components. What is the standard deviation of their dot product? (b) Sixty-four independent contributions, each of variance 0.04, accumulate at one point. What is the resulting standard deviation?
(a) Variance \( = 288 \), standard deviation \( = \sqrt{288} = 12\sqrt{2} \approx 17.0 \). (b) Variance \( = 64 \times 0.04 = 2.56 \), standard deviation \( = 1.6 \).
Whenever something can go one of two ways — a residual block is traversed or skipped, a reasoning chain is right or wrong, a token is correct or not — and the trials are independent, the count of “successes” follows a binomial distribution. You need three things about it: its shape, its summary statistics, and how to approximate it with a bell curve when the counts get large.
For \( n \) independent trials each succeeding with probability \( p \), the probability of exactly \( k \) successes is \( \binom{n}{k} p^k (1-p)^{n-k} \). The mean number of successes is \( np \) and the variance is \( np(1-p) \); take the square root of the latter for the standard deviation. When \( n \) is large, the distribution is well approximated by a normal curve with that same mean and standard deviation, and probabilities over a range of counts become areas under the bell — read off from the standard normal function \( \Phi \). Because the binomial is discrete and the normal continuous, we widen the target interval by half a unit at each end; this continuity correction is the little half-step that keeps the approximation honest.
\( \Phi(z) \) is the probability that a standard normal variable falls below \( z \), and a symmetric interval \( [-z, +z] \) therefore holds \( 2\Phi(z) - 1 \). You will need a few of its values with no table to hand, so here they are.
\( \Phi \): 0.5 → 0.691 · 1.0 → 0.841 · 1.5 → 0.933 · 2.0 → 0.977 · 2.04 → 0.9793 · 2.1 → 0.9821 · 2.5 → 0.994 · 3.0 → 0.9987
Interpolate linearly between listed points; the error lands in the fourth decimal, below anything the arithmetic in this book can notice. The two awkward entries, \( \Phi(2.04) \) and \( \Phi(2.1) \), are spelled out because a residual-network result in Chapter 4 turns on them and you will be asked to reproduce it closed-book.
Majority vote of five
Five independent reasoning chains each reach the correct answer with probability \( 0.6 \); the majority answer is taken. What is the probability the majority is correct? Here \( n = 5 \) is too small for the normal approximation, so we sum the binomial directly for \( k \ge 3 \).
\( P(3) = \binom{5}{3}(0.6)^3(0.4)^2 = 10 \times 0.216 \times 0.16 = 0.3456 \). \( P(4) = \binom{5}{4}(0.6)^4(0.4) = 5 \times 0.1296 \times 0.4 = 0.2592 \). \( P(5) = (0.6)^5 = 0.0778 \). The sum is \( 0.683 \). Sampling five chains and voting lifts a \( 60\% \) chain to a \( 68\% \) answer — and Chapter 15 shows this only ever helps when the single-chain probability already exceeds one half.
Five independent chains are each correct with probability \( 0.7 \). What is the probability that the majority of the five is correct?
\( P(3) = 10(0.343)(0.09) = 0.3087 \); \( P(4) = 5(0.2401)(0.3) = 0.3602 \); \( P(5) = 0.16807 \). Sum \( = 0.837 \).
The most valuable single calculation in this book is this: minimize a quantity that depends on two variables, when those variables are tied together by a constraint. The compute-optimal training recipe of Chapter 7 — arguably the most consequential result in modern machine learning — is exactly this calculation, and so are several smaller optimizations along the way. The method is humble: use the constraint to eliminate one variable, then minimize an ordinary one-variable function by setting its derivative to zero.
Concretely, suppose you must minimize \( f(N, D) \) subject to a constraint of the form \( ND = C \) for a fixed budget \( C \). Solve the constraint for one variable, \( D = C/N \), substitute to obtain a function of \( N \) alone, differentiate, and set the derivative to zero. The value of \( N \) you find, together with \( D = C/N \), is the constrained optimum. The subtlety that trips people — and it is worth previewing now so it is not a surprise in Chapter 7 — is that the balance condition at the optimum usually involves the exponents of the terms, not merely the terms themselves. Substitution makes that fall out automatically, which is exactly why we prefer it to guessing.
A constrained minimum, start to finish
Minimize \( f = N^{-1} + 4D^{-1} \) subject to \( ND = 100 \).
Substitute \( D = 100/N \): \( f(N) = N^{-1} + 4 \cdot \frac{N}{100} = \frac{1}{N} + \frac{N}{25} \). Differentiate and set to zero: \( f'(N) = -N^{-2} + \frac{1}{25} = 0 \), so \( N^2 = 25 \), \( N = 5 \). Then \( D = 100/5 = 20 \), and \( f = 0.2 + 0.2 = 0.4 \). Notice that the two terms came out equal at the optimum — a recurring signature you can often use to check your work, though in Chapter 7 the equal quantities will be the exponent-weighted terms rather than the bare ones.
The same reflex, in a slightly different guise, handles stability questions. When an iterative process multiplies an error by a fixed factor each step, it converges precisely when the magnitude of that factor is below one. Reading off the condition “\( |1 - \eta\lambda| < 1 \)” and solving it for the allowed range of \( \eta \) is the opening move of Chapter 3. It is the same comfort with a one-variable inequality that substitution demands, so we practice it here.
(a) Minimize \( 2N^{-1} + D^{-1} \) subject to \( ND = 50 \). (b) For which positive step sizes \( \eta \) does the iteration \( z \leftarrow (1 - \eta\lambda)z \) drive \( z \) to zero, given \( \lambda > 0 \)?
(a) \( D = 50/N \) gives \( f = 2/N + N/50 \); \( f' = -2/N^2 + 1/50 = 0 \Rightarrow N^2 = 100, N = 10 \), \( D = 5 \), \( f = 0.2 + 0.2 = 0.4 \). (b) Need \( |1 - \eta\lambda| < 1 \), i.e. \( 0 < \eta\lambda < 2 \), so \( 0 < \eta < 2/\lambda \).
An effect that repeats at every future step, discounted by a constant factor each time, sums to a geometric series. Two closed forms cover every use in this book. Starting from the present step, \( \sum_{t \ge 0} \gamma^t = \frac{1}{1-\gamma} \) for \( |\gamma| < 1 \); starting from the next step, \( \sum_{t \ge 1} \gamma^t = \frac{\gamma}{1-\gamma} \). That is all. Chapter 10 uses the second form to weigh a stream of future rewards against a single reward now, and to derive the threshold at which an agent would rather seize a small perpetual reward than accept a large one-off — the arithmetic behind a famous cautionary tale about misaligned incentives.
Perpetual trickle versus one-time prize
An agent values future rewards with discount factor \( \gamma = 0.95 \) per step. Option A pays 1 unit now and nothing after. Option B pays \( r \) units at every step from the next onward. For which \( r \) does the agent prefer B?
Option B is worth \( \sum_{t \ge 1} \gamma^t r = \frac{\gamma r}{1-\gamma} = \frac{0.95\,r}{0.05} = 19r \). This exceeds Option A's value of 1 when \( r > 1/19 \approx 0.053 \). A perpetual reward of barely five percent of the one-time prize is already preferable — a small fact with large consequences in Chapter 10.
With discount factor \( \gamma = 0.9 \), what perpetual per-step reward \( r \) (paid from the next step onward) is worth exactly 2 units of present reward?
Value \( = \frac{0.9\,r}{0.1} = 9r = 2 \Rightarrow r = 2/9 \approx 0.22 \).
The final reflex is to measure information in bits, because Chapter 10 — and, quietly, much of the rest of the book — treats learning as compression. Three facts carry the load. A choice among \( M \) equally likely options costs \( \log_2 M \) bits to specify. A set of code words with lengths \( \ell_i \) can be made unambiguously decodable if and only if \( \sum_i 2^{-\ell_i} \le 1 \) — the Kraft inequality — which we will read in reverse as a rule that assigns a probability \( 2^{-\ell} \) to a description of length \( \ell \). And the number of ways to choose half of \( 2n \) items, \( \binom{2n}{n} \), has a logarithm close to \( 2n - \tfrac12 \log_2(\pi n) \); you will be given this approximation whenever it is needed, but you should know what it says — that a balanced binary pattern is very nearly incompressible.
The reason to install this now, rather than in Chapter 10, is that the interpretation matters more than the arithmetic. When a model assigns probabilities to the next token, the number of bits it would take to encode the true token is \( -\log_2 \) of the assigned probability; the average of that quantity is the cross-entropy; and a lower cross-entropy is literally a shorter encoding of the data. “A better predictor is a better compressor” is not a metaphor in this book — it is an identity you will compute.
A probability from a code length
Two descriptions remain consistent with some data: one is 3 bits long, the other 5 bits. Reading the Kraft inequality as a prior, we assign each a weight \( 2^{-\ell} \): the first gets \( 2^{-3} = 1/8 \), the second \( 2^{-5} = 1/32 \). Normalizing, the shorter description holds \( \frac{1/8}{1/8 + 1/32} = \frac{4}{5} = 0.8 \) of the belief. Every additional bit of length halves a description's share — which, as Chapter 10 shows, is Occam's razor stated as a theorem rather than a preference.
(a) How many bits are needed to single out one option from 1000 equally likely ones? (b) Three consistent descriptions have lengths 2, 4, and 4 bits. Under the \( 2^{-\ell} \) prior, what share of belief does the shortest hold?
(a) \( \log_2 1000 = \ln 1000/\ln 2 = 6.908/0.693 \approx 9.97 \) bits — just under 10. (b) Weights \( 2^{-2}=1/4 \), \( 2^{-4}=1/16 \), \( 2^{-4}=1/16 \); total \( = 4/16 + 1/16 + 1/16 = 6/16 \). Shortest share \( = (4/16)/(6/16) = 2/3 \).
Two probabilities that look almost the same are not the same, and the gap between them is where a great deal of confident wrong reasoning about models lives. The chance that a leaked question is answered correctly is one number. The chance that a correct answer came from a leaked question is another. Getting from the first to the second is one line of algebra.
Write \( P(H \mid E) \) for the probability of a hypothesis \( H \) given evidence \( E \). Bayes' rule turns one conditional into the other:
\[ P(H \mid E) = \frac{P(E \mid H)\,P(H)}{P(E)}, \]and the denominator is just the total probability of the evidence, assembled from every way it could arise: \( P(E) = \sum_i P(E \mid H_i)P(H_i) \) over a set of hypotheses that covers all the cases. In this book the sum almost always has two terms, and the whole calculation fits on one line.
Before working an example, notice what the formula says about magnitudes, because the surprise it produces is the reason the reflex is worth installing. If a hypothesis is rare, then even evidence that follows from it almost certainly can leave it unlikely — the small \( P(H) \) on top is not rescued by a large \( P(E \mid H) \). Most people's instinct runs the other way, and reads strong evidence as a strong conclusion regardless of how rare the cause was to begin with.
Which successes were memory
A quarter of a test set was seen during training and those items are always answered correctly; on the remaining three quarters the model answers correctly with probability \( 0.6 \). A question is answered correctly. What is the probability it was one of the seen ones?
Take \( H \) to be “this item was seen”, so \( P(H) = 0.25 \) and \( P(E \mid H) = 1 \). For an unseen item, \( P(E \mid \text{not } H) = 0.6 \). Assemble the denominator first:
\[ P(E) = 1 \times 0.25 + 0.6 \times 0.75 = 0.25 + 0.45 = 0.70. \]Then Bayes' rule gives \( P(H \mid E) = (1 \times 0.25)/0.70 = 0.357 \). Better than a third of this model's correct answers are recall rather than skill — and note that the observed accuracy, \( 0.70 \), fell out of the denominator on the way. Chapter 9 runs this calculation backwards, inferring the quarter from the seventy percent, and the algebra is the same three symbols.
(a) One tenth of a benchmark leaked and is always answered correctly; the clean accuracy is \( 0.5 \). Give the observed accuracy, and the probability that a correct answer was leaked. (b) A rare failure mode occurs in one run in a thousand. A detector fires on every genuine occurrence and also on 2% of healthy runs. The detector fires. What is the probability the failure is real?
(a) \( P(E) = 0.1 + 0.5(0.9) = 0.55 \); \( P(H \mid E) = 0.1/0.55 = 0.18 \). (b) \( P(E) = 1(0.001) + 0.02(0.999) = 0.001 + 0.020 = 0.021 \), so \( P(H \mid E) = 0.001/0.021 = 0.048 \). A detector that never misses still leaves you wrong nineteen times in twenty when it fires, because the thing it looks for is rare. This is the magnitude surprise above, in its usual costume.
The last reflex is the cheapest to state and the most used in Part III: knowing what a computation costs before you run it. Everything expensive in this book is a matrix product, and a matrix product has exactly one cost formula.
The matrix-multiply primitive is the whole of it. Multiplying an \( a \times b \) matrix by a \( b \times c \) matrix produces \( ac \) entries, each a sum of \( b \) products, so the cost is
\[ a\,b\,c \quad \text{multiply--accumulate operations,} \]where a multiply–accumulate is one multiplication and one addition taken together, as the hardware performs it. Count these, not multiplications and additions separately, and the arithmetic in later chapters will match the figures the field quotes. When one of the three dimensions is 1 — a matrix meeting a single vector — the formula still holds and the cost collapses to \( ab \), a fact Chapter 11 builds an entire economics on.
One corollary carries more weight than the rule itself. Every weight in a matrix is touched exactly once per input the matrix processes, so a layer of \( P \) weights costs about \( P \) multiply–accumulates per input. Cost per input is therefore a statement about parameter count, and nothing else — which is why so much of the field's arithmetic can be done knowing only how big a model is.
The cost of one feed-forward block
A layer takes a vector of width \( 512 \), maps it up to \( 2048 \), and maps it back down to \( 512 \). What does it cost at one position, and how does that compare with its parameter count?
Up: a \( 1 \times 512 \) vector against a \( 512 \times 2048 \) matrix, so \( 1 \times 512 \times 2048 = 1{,}048{,}576 \) multiply–accumulates. Down: \( 1 \times 2048 \times 512 \), the same again. The block costs \( 2{,}097{,}152 \) operations per position, about \( 2.1 \) million.
Its weights number \( 512 \times 2048 + 2048 \times 512 = 2{,}097{,}152 \) — the same figure, as the corollary promised. Write the width as \( d \) and the middle as \( 4d \) and this is \( 8d^2 \) either way, which is the expression Chapter 6 balances against the cost of attention.
(a) Give the multiply–accumulate cost of multiplying a \( 128 \times 512 \) matrix by a \( 512 \times 64 \) matrix. (b) A model holds \( 7 \times 10^9 \) weights. Roughly how many multiply–accumulates does it perform to process one input? (c) Two square matrices of side \( n \) are multiplied. How does the cost grow when \( n \) doubles?
(a) \( 128 \times 512 \times 64 = 4{,}194{,}304 \), about \( 4.2 \) million. (b) About \( 7 \times 10^9 \), one per weight. (c) The cost is \( n^3 \), so doubling \( n \) multiplies it by eight.
That is the whole toolkit. Logarithms turn every power law into a straight line and every compounding factor into a step count. Variance algebra tells you how independent quantities accumulate — the single most reused fact in the book. The binomial and its normal approximation handle anything that can go one of two ways. Constrained minimization by substitution is the engine behind the field's most consequential result. Geometric series price anything that repeats forever at a discount. Counting in bits is the currency in which prediction, compression, and simplicity all turn out to be the same quantity. Turning a conditional around separates what a number says from what it is usually taken to say. And counting operations converts a description of a model into a bill.
You will not meet these again as objects of study. From here they are furniture, assumed silently in every derivation, and the hours spent making them automatic now are repaid many times over. What Chapter 2 adds is not more mathematics but the objects the mathematics will be applied to — a layer, a gradient step, a softmax, a token — each written down once, precisely, so that no later derivation has to describe its own subject in words.
Work these on paper. The A set drills the reflexes; the B set combines them the way later chapters will; the C challenge points forward. Solutions to B and C are in Appendix D.
A · Drills
B · Problems
C · Challenge
Closed book, fifty minutes. Attempt all nine A-set exercises and B-1 through B-4. You pass at eight of thirteen fully correct. If you fall short, the fix is never to reread — it is to redo the drills in the section whose reflex failed, then retake the gate. These eight reflexes do not appear again as objects of study; from here they are simply assumed, and every hour spent making them automatic now is repaid tenfold in the chapters ahead.