Eight builds. Each is accepted only when it produces a number your derivation predicted in advance — because a derivation confirmed by your own running code is learned twice.
Reading a derivation convinces you. Writing code that reproduces its number to three decimal places convinces you differently, and more permanently.
Each lab is anchored to the chapter whose derivation supplies its acceptance number, so they cluster where Part III does its densest work rather than spreading one to a chapter. Each takes four to six hours. The rule that makes them worth the time: predict the number before you run anything. Write the expected value in your notebook, then run the code. When they disagree, one of the two is wrong, and finding out which is where the learning happens — more often than you would expect, it is the code.
Scaffolding: a minimal transformer implementation such as nanoGPT and the Zero to Hero lecture series cover Labs 1 through 4 comfortably. minbpe is a reference for Lab 1. From Lab 6, a preference-training library such as TRL is permitted, but write the loss function yourself first.
Implement byte-pair encoding: count adjacent pairs, merge the most frequent, repeat. Train it on a megabyte of text and use it to encode and decode round-trip.
Round-trip encoding and decoding returns the input exactly, for a corpus containing multi-byte characters. Report the compression ratio — bytes in, tokens out — and reproduce a known merge sequence on a short test string. Predict the vocabulary size after \( k \) merges before running.
Implement scaled dot-product attention and a full transformer block from primitives. Then, separately, implement the streaming softmax of Derivation 12.1 — running maximum, running normalizer, block-wise rescaling.
Your streaming softmax matches a library softmax to within \( 10^{-6} \) on random inputs of at least a thousand elements, processed in at least five blocks. Before running, compute by hand the two-block example of Derivation 12.1 and confirm your implementation reproduces 1.553.
Train a small transformer on a few megabytes of text until the validation loss is stable. Report loss in nats per character, then convert to bits per character.
Validation loss at or below 1.6 nats per character. Then the real criterion: using Chapter 10's identity, predict the compressed size of the corpus from your bits-per-character figure, write the prediction down, and compare against what a general-purpose compressor achieves on the same file. Explain any gap.
Add incremental decoding with a KV cache to your Lab 3 model, so that generating token \( t+1 \) reuses the keys and values computed for tokens \( 1 \ldots t \).
Generated output is bit-identical with and without the cache, given the same random seed — this is the correctness test, and it catches nearly every indexing bug. Then measure tokens per second with and without, and explain the speedup using Chapter 11's roofline argument. Predict the cache's size in bytes from Derivation 11.1 and verify against measured memory.
Implement a low-rank adapter on one weight matrix of your model: freeze the original, add a trainable \( BA \) product of rank \( r \), and train only the adapter.
Your printed count of trainable parameters equals \( 2rd \) exactly — not approximately. Measure optimizer-state memory before and after, and confirm the ratio matches the fraction of parameters made trainable. State which budget you saved and which you did not.
Take a small open pretrained model as both policy and frozen reference. Implement the collapsed preference loss of Derivation 14.2 yourself — the log-ratio difference, scaled by \( \beta \), through a logistic. Train on a public preference dataset.
The loss at step zero equals \( \ln 2 = 0.693 \). Derive why before you run it — at initialization the policy is the reference, both log-ratios vanish, the margin is zero, and \( -\ln\sigma(0) = \ln 2 \). This single check catches almost every implementation error. Then: the mean margin increases monotonically over training, and implied-reward accuracy on held-out pairs exceeds 60%.
Quantize your Lab 3 model's weights to 8 bits with a uniform quantizer, and measure both the weight error and the effect on model quality.
Measured mean squared quantization error is within a factor of two of the \( \Delta^2/12 \) prediction — compute the prediction first. Report the change in validation perplexity, and separately report the error for a layer with outlier weights, confirming Derivation 13.1's \( R^2 \) sensitivity.
Train or obtain a smaller draft model compatible with your Lab 3 model's tokenizer. Implement draft-and-verify decoding with the rejection rule of Derivation 12.2.
Two criteria. Correctness: over many samples at fixed temperature, the output token distribution is statistically indistinguishable from the target model's alone — the technique is exact, and your implementation must demonstrate it. Performance: measure the empirical acceptance rate \( \alpha \), then confirm that measured tokens per cycle falls within 10% of \( \frac{1-\alpha^{k+1}}{1-\alpha} \).
They are not for learning a framework. They are for closing the loop between a symbol and a measurement — so that \( \Delta^2/12 \) stops being a formula you can restate and becomes a number you have watched appear on your own screen. Three of the eight acceptance criteria (Labs 2, 6, and 8) are exactness checks: they verify that a technique you derived as mathematically equivalent really is equivalent in floating point. Those three are the most valuable, because exactness is the property most often assumed and least often verified.
If time is short, do Labs 2, 4, and 6. They cover the streaming derivation, the memory-versus-compute distinction that organizes all of Part III, and the collapse that organizes modern post-training.