Seven things this book computes with, written down exactly once. Not what they are for — only what they are.
You cannot derive a result about a thing you have not written down. What follows is the shortest possible statement of every object the rest of this book manipulates: a network, a gradient step, a softmax, a Jacobian product, a token, a divergence, a normalizer.
This is a reference chapter and it is deliberately flat. There are no surprises in it and nothing here is argued for — the arguing starts in Chapter 3, and every one of these objects earns its place there. Read it once at normal speed, then use it the way you use a table of integrals: when a symbol in a later chapter stops meaning something, come back, and go away again.
One warning about what this chapter is not. It does not explain why neural networks work, what problems they solve, or how anyone came to write them down in this form. Those are excellent questions and this is the wrong book for them; the readings at the end are the right ones. The book you are holding assumes you have seen a network before and asks a different question — what does it cost, and what does the arithmetic force.
A layer takes a vector \( x \) of width \( m \), forms weighted sums of its entries, adds a constant to each, and passes the result through a fixed nonlinear function \( \phi \) applied entrywise:
\[ y = \phi(Wx + b), \qquad W \in \mathbb{R}^{n \times m},\ b \in \mathbb{R}^{n}. \]\( W \) holds the weights and \( b \) the biases; together they are the layer's parameters, and Chapter 1's counting rule says the layer costs about \( nm \) multiply–accumulates per input. The entries of \( y \) are the layer's activations. A network stacks such layers, each consuming the previous one's output; running an input through the stack is the forward pass. Write \( \theta \) for all the parameters of all the layers collected into one vector, and \( N \) for how many numbers that is — the quantity Chapter 7 spends its entire budget on.
The commonest nonlinearity in this book is the rectifier, \( \phi(z) = \max(0, z) \): zero for negative input, the identity for positive. It contributes no gradient where it is flat, a fact Chapter 3 turns into a practical warning about biases.
A loss \( L(\theta) \) is a single number measuring how badly the network does on data, and training means making it smaller by changing \( \theta \). The only search procedure that scales is gradient descent: compute the gradient \( g = \nabla_\theta L \), which points in the direction of steepest increase, and step against it,
\[ \theta \leftarrow \theta - \eta\, g, \]where the step size \( \eta \) (also called the learning rate) is a small positive number. Chapter 3 is about how badly this can go.
Momentum is one modification, and because Chapter 3 derives a result about it that you will be asked to reproduce three times, it is worth having the update itself in front of you. Keep a running average \( v \) of recent gradients and step along that instead:
\[ v \leftarrow \beta v + g, \qquad \theta \leftarrow \theta - \eta\, v, \]with \( \beta \) between 0 and 1 — typically 0.9. Where successive gradients agree, \( v \) accumulates and the effective step grows; where they alternate in sign, they cancel. That is the whole mechanism, and Chapter 3 prices it.
(a) A layer maps width 1024 to width 4096 with biases. How many parameters, and how many multiply–accumulates per input? (b) In the momentum update with \( \beta = 0.9 \), a constant gradient \( g \) is supplied at every step. What value does \( v \) approach?
(a) \( 4096 \times 1024 + 4096 = 4{,}198{,}400 \) parameters; about \( 4.19 \) million multiply–accumulates, one per weight. (b) \( v \to g(1 + \beta + \beta^2 + \cdots) = g/(1-\beta) = 10g \) — Chapter 1's geometric series, and the reason momentum takes effectively larger steps along a consistent direction.
When a network must choose among \( V \) options it emits a vector of \( V \) real numbers called logits, written \( z \). Logits are unconstrained: any real values, positive or negative. The softmax converts them into a probability distribution by exponentiating and normalizing:
\[ \operatorname{softmax}(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{V} e^{z_j}}. \]Three properties are used constantly and none is deep. The outputs are positive and sum to one, so this is a distribution. Adding the same constant to every logit changes nothing, because the constant factors out of numerator and denominator alike — softmax sees only differences, which is why Chapter 12's streaming version may subtract a running maximum without approximation. And when one logit exceeds the others by much more than 1, its probability approaches 1 and the rest collapse toward zero; the distribution saturates, and a saturated softmax has almost no gradient, since moving the logits barely moves the probabilities. Chapter 6 rescales attention scores for exactly this reason.
With two options the softmax reduces to the logistic function of the logit difference,
\[ \sigma(u) = \frac{1}{1 + e^{-u}}, \]which recurs throughout the book as a gate value in Chapter 5 and as a preference probability in Chapter 14. Note \( \sigma(0) = \tfrac12 \), so \( -\ln\sigma(0) = \ln 2 = 0.693 \) — a number that turns out to be an acceptance test for a piece of code in Chapter 14.
Given a true outcome \( i \) and a predicted distribution \( q \), the cross-entropy loss is \( -\ln q_i \), and its average over a dataset is the loss almost every model in this book minimizes. Chapter 1 already noted that \( -\log_2 q_i \) is the number of bits an optimal code spends on outcome \( i \); the loss is therefore a code length, which is the identity Chapter 10 builds a chapter on. Perplexity is the same quantity re-expressed as an effective number of equally likely options, \( 2^{H} \) with \( H \) the cross-entropy in bits.
One prediction, priced
A model with a vocabulary of four sees a context and emits logits \( (3.0,\ 1.0,\ 0.5,\ 0.0) \). The token that actually follows is the second. What does that prediction cost, in nats, in bits, and as a perplexity?
Exponentiate: \( e^{3} = 20.09 \), \( e^{1} = 2.718 \), \( e^{0.5} = 1.649 \), \( e^{0} = 1 \), summing to \( 25.45 \). The distribution is \( (0.789,\ 0.107,\ 0.065,\ 0.039) \). The true token was assigned \( q = 0.107 \), so the loss is \( -\ln 0.107 = 2.237 \) nats, which is \( 2.237/0.693 = 3.23 \) bits, and the perplexity is \( 2^{3.23} = 9.4 \).
Read the last number carefully, because it is the one people misread. A perplexity of 9.4 on a vocabulary of 4 does not mean the model was choosing among nine options — there were only four. It means this particular prediction was worse than guessing uniformly, which would have cost \( \log_2 4 = 2 \) bits. Perplexity is an effective branching factor, and nothing stops it exceeding the real one when the model is confidently wrong. That is exactly what happened here: 0.789 of the mass went somewhere else.
(a) Logits are \( (2, 0, -1) \). Give the softmax probabilities. (b) Add 5 to every logit and give them again. (c) A model assigns the true token probability \( 0.25 \). Give the cross-entropy loss in nats and in bits.
(a) \( e^2 = 7.389 \), \( e^0 = 1 \), \( e^{-1} = 0.368 \); total \( 8.757 \). Probabilities \( (0.844, 0.114, 0.042) \). (b) Identical — the shared factor \( e^5 \) cancels. (c) \( -\ln 0.25 = 1.386 \) nats \( = -\log_2 0.25 = 2 \) bits.
A network is a composition of functions, so its derivatives are governed by the chain rule. When the functions map vectors to vectors, the object playing the role of a derivative is the Jacobian: for \( y = F(x) \) with \( x \in \mathbb{R}^m \) and \( y \in \mathbb{R}^n \), the Jacobian \( J \) is the \( n \times m \) matrix whose \( (i,j) \) entry is \( \partial y_i / \partial x_j \). It says how each output responds to each input.
The chain rule then reads as a matrix product. For a stack of \( \ell \) transformations, the sensitivity of the last output to the first input is
\[ \frac{\partial h_L}{\partial h_0} = J_L\,J_{L-1}\cdots J_1, \]and this product — a long chain of matrices multiplied together — is the single most consequential object in Part II. If the factors shrink, the product vanishes geometrically with depth; if they grow, it explodes. Chapter 4 fixes the first problem by adding an identity to every factor, and Chapter 5 meets the same product running along time instead of depth.
Backpropagation is the name for evaluating this product efficiently: sweep forward storing activations, then sweep backward multiplying by one Jacobian at a time, accumulating the gradient with respect to each layer's parameters as you pass it. Two facts about it are used later and neither requires the algorithm's details. The backward sweep costs roughly twice the forward sweep, for the reason the next box counts. And the activations from the forward pass must be kept until the backward sweep reaches them, which is why they occupy memory — the third of the budgets Chapter 13 names.
Six operations per parameter per token
The two facts above are enough to price training, and the number they produce carries four later chapters. Work in floating-point operations rather than multiply–accumulates, since that is the unit accelerators are sold in: one multiply–accumulate is a multiply and an add, so it is 2 FLOPs.
Forward. By Chapter 1's corollary, each of the \( N \) parameters is used in one multiply–accumulate per token. That is \( N \) multiply–accumulates, or \( 2N \) FLOPs per token.
Backward. Each layer must produce two gradients, not one: the gradient with respect to its input, to hand to the layer below, and the gradient with respect to its own weights. Each is a product against the same weight matrix and costs what the forward pass cost. So the backward sweep is \( 2N \) multiply–accumulates, or \( 4N \) FLOPs per token — twice the forward pass, which is the origin of that ratio.
Adding them, training costs about \( 6N \) FLOPs per token, so a run over \( D \) tokens costs
\[ C \approx 6ND \ \text{FLOPs}, \]and serving, which performs the forward pass alone, costs about \( 2N \) FLOPs per generated token. The 6 and the 2 are not measurements of anything. They are 2 FLOPs per multiply–accumulate times one, two or three passes over the weights, and Chapters 7, 11 and 17 spend their entire budgets in these units.
(a) A 13-billion-parameter model is trained on 2 trillion tokens. Give the training cost in FLOPs. (b) The same model serves \( 5\times10^{12} \) tokens over its life. Give the serving cost, and its ratio to training. (c) Why is the backward pass twice the forward one rather than the same or three times?
(a) \( 6 \times 1.3\times10^{10} \times 2\times10^{12} = 1.56\times10^{23} \) FLOPs. (b) \( 2 \times 1.3\times10^{10} \times 5\times10^{12} = 1.3\times10^{23} \), about \( 0.83 \) times the training cost — serving a popular model is the same order of expense as building it, which is the whole of Chapter 17. (c) Because two gradients are needed at each layer and only one activation: one gradient flows down to the previous layer, one lands on the weights. A layer at the very bottom needs only the second, which is why 6 is an approximation slightly on the high side.
Language models do not consume characters or words. Text is first cut into tokens, drawn from a fixed vocabulary of size \( V \), typically tens of thousands of entries: common words are single tokens, rare words split into several pieces, and every possible string is representable because single bytes are in the vocabulary as a fallback. The number of tokens a corpus becomes is what the symbol \( D \) counts in Chapter 7, and the compression ratio — characters per token, usually around four for English — is what converts between a corpus's size on disk and its size in the units this book budgets in.
A token is turned into a vector by lookup in an embedding matrix of shape \( V \times d \), where \( d \) is the model's width; one row per vocabulary entry. At the other end, a readout matrix of shape \( d \times V \) turns the final activations back into \( V \) logits. When the two matrices are the same array used twice they are called tied, which saves \( Vd \) parameters; when they are separate they are untied. Both tables scale with vocabulary rather than with depth, which is why Chapter 5 finds them occupying nearly half of a small model and why scaling studies count parameters excluding them.
A model has \( d = 4096 \) and \( V = 32{,}000 \), with untied embedding and readout. (a) How many parameters sit in the two tables together? (b) A corpus of 40 GB of English text is tokenized at four characters per token; roughly how many tokens is that?
(a) \( 2 \times 32{,}000 \times 4096 = 262 \) million. (b) Roughly \( 4\times10^{10}/4 = 10^{10} \) tokens, ten billion — assuming one byte per character, which for English is close enough.
A normalization layer takes a vector, subtracts its mean, divides by its standard deviation, and then applies a learned scale and shift:
\[ \operatorname{norm}(x)_i = \gamma_i\,\frac{x_i - \mu}{\sqrt{s^2 + \epsilon}} + \beta_i, \qquad \mu = \frac{1}{d}\sum_j x_j,\ \ s^2 = \frac{1}{d}\sum_j (x_j - \mu)^2, \]with \( \epsilon \) a small constant guarding against division by zero, and \( \gamma, \beta \) learned vectors of width \( d \). The point is that whatever the layer receives, what it emits has a controlled scale, set by \( \gamma \) rather than by whatever happened upstream. Variants differ in what they average over and in whether they bother to subtract the mean; none of those distinctions matters to any calculation in this book. What matters is where such a layer is placed, which Chapter 4 settles by an argument about accumulated variance.
Two distributions over the same set can be compared by a single number. The Kullback–Leibler divergence from \( q \) to \( p \) is
\[ \mathrm{KL}(p \,\|\, q) = \sum_i p_i \ln\frac{p_i}{q_i} \quad \text{nats}, \]the average, under \( p \), of the log-ratio between the two. Read it as a cost: it is the number of extra nats per outcome you pay for encoding data that really follows \( p \) using a code built for \( q \). Three properties are all that later chapters use. It is never negative. It is zero exactly when \( p = q \). And it is not symmetric — \( \mathrm{KL}(p\|q) \) and \( \mathrm{KL}(q\|p) \) are different numbers, so the order of the arguments is never decorative.
Chapter 10 evaluates this for two Gaussians and reads the answer as the price of a weight's precision. Chapter 14 uses it as a leash, penalizing a policy for drifting from the one it started as. Both are the same formula with different distributions substituted in.
Let \( p = (0.5, 0.5) \) and \( q = (0.9, 0.1) \). Compute \( \mathrm{KL}(p\|q) \) and \( \mathrm{KL}(q\|p) \) in nats, and confirm they differ.
\( \mathrm{KL}(p\|q) = 0.5\ln(0.5/0.9) + 0.5\ln(0.5/0.1) = 0.5(-0.588) + 0.5(1.609) = 0.511 \). \( \mathrm{KL}(q\|p) = 0.9\ln(0.9/0.5) + 0.1\ln(0.1/0.5) = 0.9(0.588) + 0.1(-1.609) = 0.368 \). Different, as promised, and both positive.
Nothing was derived here, and that is the point: these seven objects are now written down, so that later chapters can compute with them instead of around them. A layer is a matrix, a bias and a nonlinearity, and its cost per input is its parameter count. Gradient descent steps against the gradient; momentum steps against a running average of gradients, and that running average is a geometric series. Softmax turns logits into probabilities, sees only their differences, and stops responding when it saturates. A network's sensitivity is a product of Jacobians, and long products are the recurring danger of Part II. Text becomes tokens, and tokens become rows of two large tables that scale with vocabulary rather than depth. Normalization fixes the scale of what leaves a layer. And a divergence prices the gap between two distributions, asymmetrically.
Chapter 3 begins the actual subject, and it begins by taking the humblest object above — the gradient step of Section 2.1 — and asking a question that held the field back for twenty years: not whether a good setting of \( \theta \) exists, but whether stepping downhill can ever reach it.
These check that the objects are in place, nothing more. If they are slow, reread the section rather than pressing on; every later chapter assumes this vocabulary silently. Solutions to B and C are in Appendix D.
A · Drills
B · Problems
C · Challenge
Closed book, twenty minutes. Write down, from memory and with every symbol named: the layer equation, the gradient-descent and momentum updates, the softmax, the cross-entropy loss, the Jacobian chain product, and the KL divergence. Then attempt A-2 and B-2. You pass when all seven are correct and complete — not approximately right. Every one of them appears inside a derivation you will be asked to reproduce later, and a symbol you cannot define is a derivation you cannot check.