Look here when a symbol stops meaning what you thought it meant. Several carry more than one meaning, because the field's own conventions collide and renaming them would make this book harder to read against the papers rather than easier.
Symbols that mean one thing throughout
- \( N \)Number of parameters in a model.Ch. 2
- \( D \)Number of training tokens.Ch. 7
- \( C \)A compute budget, in floating-point operations.Ch. 7
- \( T \)Tokens generated over a deployment's lifetime.Ch. 17
- \( \theta \)All parameters of a network, collected into one vector.Ch. 2
- \( \eta \)Step size, also called the learning rate.Ch. 2
- \( \lambda \)A curvature of the loss surface along one axis.Ch. 3
- \( H \)Number of attention heads; \( H_{kv} \) when keys and values are shared.Ch. 6, 11
- \( V \)Vocabulary size.Ch. 2
- \( s \)Sequence position, or sequence length.Ch. 11
- \( \Phi \)The standard normal cumulative distribution; values in Section 1.3.Ch. 1
- \( \pi \)A policy: the model as a distribution over responses. \( \pi_{\text{ref}} \) is the frozen starting copy.Ch. 14
- \( Z \)A partition function — whatever a non-negative quantity must be divided by to sum to one.Ch. 14
- \( \Delta \)Quantizer step size.Ch. 13
- \( \Upsilon \)The complexity-weighted intelligence measure.Ch. 10
Symbols that carry more than one meaning
Each is redefined at first use in every chapter that uses it. If one appears without a definition nearby, that is a defect — report it.
- \( \beta \)Five meanings. A power-law exponent (Ch. 1, 5); the momentum coefficient (Ch. 2, 3); bytes per stored value (Ch. 11); the strength of the KL leash (Ch. 14); the per-step noise rate of a diffusion schedule (Ch. 16). The last two are adjacent and are the pair to watch.
- \( k \)Seven meanings. Kernel side (Ch. 4); the constant in \( C = kND \), which is 6 (Ch. 7); number of independent parts of a task (Ch. 9); draft length (Ch. 12); the \( k \) of top-\( k \) routing (Ch. 13); number of samples drawn (Ch. 15); sampling stride (Ch. 16).
- \( \sigma \)A weight's standard deviation (Ch. 1, 3); the logistic function \( \sigma(u) = 1/(1+e^{-u}) \) (Ch. 2, 5, 14); coding widths \( \sigma_p, \sigma_q \) and a noise standard deviation (Ch. 10).
- \( \alpha \)A loss exponent (Ch. 7); the acceptance rate of a draft token (Ch. 12); the noise-retention factor, with \( \bar\alpha_t \) its running product (Ch. 16).
- \( \epsilon \)A tolerance (Ch. 1, 5); the clip width of the surrogate (Ch. 14); the noise vector a diffusion model predicts (Ch. 16).
- \( A, B \)Coefficients of the loss surface (Ch. 7); the low-rank factors of an adapter (Ch. 13); \( A \) alone is an advantage (Ch. 14); \( B \) alone is a batch size (Ch. 11). Also option labels in comparisons.
- \( d \)Model width; \( d_{\text{head}} = d/H \) is the width one head works in, and it is \( d_{\text{head}} \) that the attention scaling divides by.
- \( L \)The loss (Ch. 2, 7); the number of layers or residual blocks (Ch. 4, 8, 11).
- \( E \)The irreducible term of the loss surface (Ch. 7); the number of experts (Ch. 13); an expectation, written \( \mathbb{E} \).
- \( p \)A probability (Ch. 9, 15); the parameter count in the OLS excess risk (Ch. 10); the target distribution in speculative decoding (Ch. 12).
- \( \gamma \)A discount factor (Ch. 1, 10); the tokenizer's compression ratio in characters per token (Ch. 2, 10).
- \( \rho \)A per-step contraction rate (Ch. 3); the probability ratio \( \pi/\pi_{\text{old}} \) (Ch. 14).
- \( \kappa \)The condition number (Ch. 3); the additive constant a reward is identified only up to (Ch. 14).
Glossary
Where each term is defined. Terms stated in Chapter 2 are the objects the book computes with; the rest are introduced where they are first needed.
- Activation§2.1 — an entry of a layer's output. Activations are retained for the backward pass, which is why they occupy memory (§8.3).
- Advantage§14.1 — a response's reward minus a baseline. Positive for better than expected.
- Arithmetic intensity§11.2 — operations performed per byte moved. Compared against the machine's ridge point to decide what binds.
- Backpropagation§2.3 — evaluating the product of Jacobians efficiently, by sweeping backward and accumulating.
- Baseline§14.1 — a prediction of a prompt's typical reward, subtracted to leave only what distinguishes responses.
- Bubble§8.1 — the fraction of accelerator-time a pipeline spends idle filling and draining.
- Checkpointing§8.3 — storing activations only at segment boundaries and recomputing the rest, at \( g = \sqrt{L} \).
- Compression ratio§2.4 — characters per token. Converts bits per token into bits per character.
- Condition number§3.1 — largest curvature over smallest. Sets how slowly gradient descent travels.
- Contamination§9.2 — the fraction of a test set seen during training.
- Coverage§15.1 — the probability that a correct answer is somewhere among \( k \) samples. Not accuracy.
- Critic§14.1 — a learned network supplying the baseline, roughly the policy's size, and the thing group-relative methods delete.
- Cross-entropy§2.2 — \( -\ln q \) for the outcome that occurred; literally a code length (§10.2).
- Decode§11.1 — generating one token at a time. Memory-bound.
- Forget gate§5.1 — the factor multiplying a recurrent cell's memory each step, and therefore its gradient horizon.
- Head§6.1 — one of \( H \) parallel attention computations, each of width \( d/H \). Splitting is free.
- Jacobian§2.3 — the matrix of partial derivatives of a vector output with respect to a vector input.
- KL divergence§2.6 — the extra cost of coding \( p \) with a code built for \( q \). Non-negative, zero only at equality, asymmetric.
- KV cache§11.1 — stored keys and values for previous positions, so they need not be recomputed. Linear in context.
- Logit§2.2 — an unconstrained score, before the softmax.
- MFU§11.3 — model FLOP utilization: useful operations per second over the machine's peak.
- Mixed precision§8.2 — half-precision arithmetic with a full-precision master copy of the weights.
- Multiply–accumulate§1.8 — one multiplication and one addition, taken together. Two FLOPs.
- Normalization§2.5 — subtract the mean, divide by the standard deviation, apply a learned scale and shift.
- Perplexity§2.2 — cross-entropy re-expressed as an effective branching factor, \( 2^H \).
- Policy§14.1 — the model seen as a distribution over responses to a prompt.
- Prefill§11.1 — processing a whole prompt at once. Compute-bound.
- Randomness deficiency§10.5 — how much shorter a member's description is than a typical member's.
- Residual connection§4.2 — adding a block's input to its output, so the Jacobian gains an identity.
- Ridge point§11.2 — a machine's peak operation rate divided by its bandwidth. Operations per byte.
- Softmax§2.2 — exponentiate and normalize. Sees only differences of logits; saturates.
- Token§2.4 — a vocabulary entry. What \( D \) counts.
- Typicality§10.5 — being a representative member of a set, as against merely belonging to it.
On the index
This book has no page index. In the web edition the browser's own search does the job better than any list. The glossary above and the derivation bank of Appendix A are the two ways in.