A problem course in modern machine learning, from the pathological valley to direct preference optimization.
Download PDF edition ↓This book teaches machine learning the way physics is taught to engineers: as a small set of results you derive, drill, and then recognize everywhere.
Most introductions to the field are surveys. They tell you that residual connections help, that larger models should see more data, that a preference-tuning method called DPO replaced a more complicated pipeline. A survey leaves you fluent in names and helpless in front of a design decision. When someone asks whether your serving cluster is memory-bound, whether a benchmark jump is capability or measurement, or why a reinforcement-learning recipe deleted its critic network, the names do not answer. A short calculation does.
The premise here is that modern machine learning — for all its scale and noise — rests on roughly forty short calculations. Each is the kind of thing a first-year engineering student could do, if someone set it up properly: a stability condition, a variance count, a constrained minimization, a geometric series, a Bayes flip. The field's landmark papers are these calculations wearing costumes. Learn the calculations and the costumes turn transparent; when next quarter's paper arrives, you will see within a page which budget it spends and which derivation prices it.
Accordingly, nothing in this book asks you to memorize a benchmark score, a release date, or an author list. Where a constant from the literature is needed — a scaling exponent, a data-law slope, an accelerator's bandwidth — it is given, the way g = 9.8 m/s² is given in a physics paper. What is never given is the model you must build to use it. That is the work, and it is the only thing examined.
The book covers two eras in one arc. Part II is the classical era, 2012–2022: why training deep networks was hard, what architecture did about it, how scale became a budgeting problem, and why compression is the master currency underneath all of it. Part III is the modern era, 2022 onward: the economics of inference, attention rebuilt around memory traffic, sparsity and thrift, the alignment pipeline, inference-time reasoning, diffusion, and the full-lifecycle cost of a deployed model. The ordering inside Part III is deliberate — systems before alignment — because without the cost vocabulary of bytes, bandwidth, and budgets, the alignment era's design choices read as fashion instead of forced moves.
Appendix A lists the thirty-eight core derivations. You are done with a chapter when you can produce its derivations on blank paper, unprompted — not when you have read it. Three of them (the compute-optimal allocation of Chapter 7, the DPO collapse of Chapter 14, and one you meet early in Chapter 3) carry a special protocol: reproduce them from nothing on three separate days before moving on.
Drills appear inside sections with their answers one click away; end-of-chapter exercise sets are graded A (drills), B (problems), and C (challenges), with full solutions in Appendix D. Keep a notebook with one line per mistake: what you did, what was right, and which trap class it belongs to. The trap taxonomy (Appendix B) names the ten ways students reliably go wrong. Twenty hours of honest error-logging is worth more than any rereading.
From Chapter 11 onward a lab manual (Appendix C) runs in parallel. Every lab's acceptance criterion is a number your derivation predicted: your streaming softmax must match a library implementation to six decimal places; your preference-tuning loss must begin at exactly ln 2 — and you must prove why before you run it. When code and derivation disagree, one of them is wrong, and finding which is the curriculum.
School-level mathematics — logarithms, elementary calculus, basic probability — and, for the labs, working Python. Chapter 1 sharpens eight mathematical reflexes and nothing else; if its gate feels easy, skip ahead without guilt. You are also assumed to know roughly what a neural network is — a stack of weighted sums and nonlinearities, trained by following a gradient. Chapter 3 states every such object precisely before it is used, but it states them; it does not motivate them.