Pretraining produces a model of what text is. Post-training bends it toward what an answer should be. The pipeline that does this collapsed, in one derivation, from a reinforcement-learning apparatus into a supervised loss.
Pretraining gives you a model of what text is. What you usually want is a model of what a good answer looks like, and the two are not the same. Getting from one to the other used to require a reward model, a reinforcement-learning loop, and a second network to estimate baselines. Then someone noticed that a term in the algebra cancels, and most of that apparatus disappeared.
We proceed in four stages. First, how human comparisons become a numerical score. Second, how a policy is trained against that score without drifting into nonsense. Third, the collapse itself, which removes most of the machinery the first two stages introduced. Fourth, a variant that deletes what remains of it, and which now underpins how reasoning models are trained.
This chapter borrows a set of terms from reinforcement learning, and they are the only place in this book where the words do more work than the mathematics. Six of them, stated once.
A policy \( \pi \) is the model itself, seen as a conditional distribution: given a prompt \( x \), it assigns a probability \( \pi(y \mid x) \) to every possible response \( y \). Nothing new is being introduced — a language model already is such a distribution, and calling it a policy only signals that we are about to change it by something other than next-token prediction. The reference policy \( \pi_{\text{ref}} \) is a frozen copy of where it started.
A reward \( r(x,y) \) is a number scoring a whole response, delivered at the end rather than per token. It may come from a model fitted to human comparisons, as in Section 14.2, or from a checker that returns 1 or 0, as in Section 14.5. Either way it is a single scalar for the entire generation, which is the feature that makes the arithmetic below simple and the credit assignment hard.
Training against a reward means raising the probability of responses that scored well. That is what a policy gradient does: sample responses from the current policy, and push up the log-probability of each in proportion to how good it was. Done naively this has a defect worth understanding, because everything else in the section is a response to it. If every response to a prompt scores between 8 and 10, the procedure pushes up all of them, and the differences between them — the only information about which was better — are swamped by their common size.
The fix is to subtract off a baseline \( b(x) \), a prediction of what this prompt typically scores, and weight by the difference. That difference is the advantage,
\[ A(x,y) = r(x,y) - b(x), \]which is positive for responses better than expected and negative for worse. Subtracting a baseline that depends only on the prompt leaves the update unbiased while shrinking its variance, which is why every method here has one. Where the baseline comes from is the whole of Section 14.5: classically a learned network of roughly the policy's size, called the critic or value network, and later just the mean of a group of samples.
Finally, samples are expensive, so the responses used for an update are usually generated by a slightly older policy \( \pi_{\text{old}} \). The probability ratio
\[ \rho = \frac{\pi(y \mid x)}{\pi_{\text{old}}(y \mid x)} \]measures how far the policy has moved on that response since it was drawn: \( \rho = 1 \) means no movement, \( \rho > 1 \) that the response has become more likely. Section 14.3 is entirely about what to do when \( \rho \) drifts too far from 1.
(a) Four responses to one prompt score \( (9, 10, 8, 9) \). Using the group mean as the baseline, give the four advantages. (b) The same four score \( (0.9, 1.0, 0.8, 0.9) \). Give the advantages, and say what a policy gradient without a baseline would do differently in the two cases.
(a) Mean 9, so advantages \( (0, +1, -1, 0) \). (b) Mean 0.9, so advantages \( (0, +0.1, -0.1, 0) \) — the same shape, ten times smaller. Without a baseline the first case would push up all four responses hard and the second gently, though the two carry identical information about which response was best. The baseline removes what the responses have in common, which is exactly the part that says nothing.
People are unreliable at scoring a response in isolation and reliable at comparing two. So the data are preference pairs: given a prompt, response \( y_w \) was preferred to \( y_l \). The standard model of such comparisons — the Bradley–Terry model — assumes an underlying scalar reward \( r \) and sets the probability of the observed preference to a logistic function of the reward difference:
\[ P(y_w \succ y_l) = \sigma\bigl(r(y_w) - r(y_l)\bigr). \]Fitting \( r \) means minimizing \( -\log\sigma(r_w - r_l) \) over the dataset. One structural fact matters enormously and is easy to miss.
Only differences are identified
The loss depends on \( r_w \) and \( r_l \) only through their difference. Adding any constant \( \kappa \) to every reward leaves \( (r_w + \kappa) - (r_l + \kappa) = r_w - r_l \) unchanged, so the loss is identical. The reward function is therefore determined only up to an additive constant — there is no meaningful “absolute” reward scale, and any claim resting on one is unfounded.
Check numerically. With \( r_w = 1.2 \) and \( r_l = 0.7 \), the loss is \( -\ln\sigma(0.5) = -\ln 0.622 = \) 0.474 nats. Shift both by \( +100 \): the difference is still \( 0.5 \) and the loss is still \( 0.474 \).
Given a reward, one could simply maximize it. One should not. A policy that maximizes a learned reward without constraint will find the regions where the reward model is wrong and exploit them — the engineering face of the reward-seizing argument from Chapter 10. The remedy is a leash: penalize divergence from the pretrained reference policy \( \pi_{\text{ref}} \). The objective becomes
\[ \max_{\pi}\ \mathbb{E}_{y\sim\pi}\bigl[r(x,y)\bigr] - \beta\,\mathrm{KL}\bigl(\pi \,\|\, \pi_{\text{ref}}\bigr), \]with \( \beta \) setting how tight the leash is. This objective — reward minus a scaled divergence from the reference — is the target for the rest of the chapter. Every method below optimizes this same objective; they differ only in how.
The classical optimizer is a policy-gradient method with a clipped surrogate, proximal policy optimization or PPO. Writing \( \rho = \pi(y)/\pi_{\text{old}}(y) \) for the probability ratio against the policy that generated the data and \( A \) for the advantage, the surrogate is
\[ \min\bigl(\rho A,\ \operatorname{clip}(\rho,\,1-\epsilon,\,1+\epsilon)\,A\bigr). \]The \( \min \) makes the objective pessimistic: it always takes the less favorable of the clipped and unclipped values, in both directions.
Where the clip kills the gradient
Take \( \epsilon = 0.2 \).
Case (i): \( \rho = 1.3 \), \( A = +2 \). Unclipped gives \( 2.6 \); clipped gives \( 1.2 \times 2 = 2.4 \). The minimum is \( 2.4 \), the clipped branch — which is constant in \( \rho \), so the gradient is zero. The policy has already moved far enough in the profitable direction; further movement earns nothing.
Case (ii): \( \rho = 0.7 \), \( A = -1 \). Unclipped gives \( -0.7 \); clipped gives \( 0.8 \times (-1) = -0.8 \). The minimum is \( -0.8 \), again the clipped branch, again zero gradient. Pessimism applies symmetrically.
For \( \rho \) inside \( [0.8, 1.2] \), clipped and unclipped agree and the gradient flows normally. The clip is a trust region enforced by making excursions unprofitable rather than forbidden.
With \( \epsilon = 0.2 \), give the surrogate value and say whether the gradient is alive, for \( A = +1 \) at \( \rho = 0.7 \), and for \( A = -1 \) at \( \rho = 1.3 \).
\( A=+1, \rho=0.7 \): unclipped \( 0.7 \), clipped \( 0.8 \); min is \( 0.7 \) — the unclipped branch, so gradient alive. \( A=-1, \rho=1.3 \): unclipped \( -1.3 \), clipped \( -1.2 \); min is \( -1.3 \) — unclipped, gradient alive. Only the two cases in Worked Example 14.1 are clipped.
Now the chapter's centerpiece. The KL-regularized objective of Section 14.3 has a closed-form optimum. Knowing it lets us eliminate the reward model entirely.
Read the next derivation slowly; it is three short moves and the third one is the whole discovery. If you have seen this result stated before, you may remember it as “the partition function is intractable, so we ignore it.” That is not what happens, and the difference is the point. The term is never ignored and never approximated. It is carried through the algebra in full and then, at the last step, it eliminates itself — for a reason you can see coming if you watch what the preference model does with two responses to the same prompt.
From a reinforcement-learning objective to a supervised loss
Move 1 — the optimum. We maximize \( \mathbb{E}_\pi[r] - \beta\,\mathrm{KL}(\pi\|\pi_{\text{ref}}) \) over all distributions \( \pi \). Write both terms as one sum and pull out \( -\beta \):
\[ \sum_y \pi(y)\,r(y) - \beta\sum_y \pi(y)\log\frac{\pi(y)}{\pi_{\text{ref}}(y)} = -\beta\sum_y \pi(y)\log\frac{\pi(y)}{\pi_{\text{ref}}(y)\,e^{\,r(y)/\beta}}. \]The denominator inside the logarithm is almost a distribution: it is non-negative, but it does not sum to one. Divide it by whatever it does sum to,
\[ Z(x) = \sum_y \pi_{\text{ref}}(y\mid x)\,e^{\,r(x,y)/\beta}, \qquad \pi^*(y\mid x) = \frac{1}{Z(x)}\,\pi_{\text{ref}}(y\mid x)\,e^{\,r(x,y)/\beta}, \]and \( \pi^* \) is a genuine distribution. Substituting it back, the objective becomes
\[ -\beta\sum_y \pi(y)\log\frac{\pi(y)}{Z(x)\,\pi^*(y)} = \beta\log Z(x) - \beta\,\mathrm{KL}(\pi\|\pi^*). \]Now read it off. The first term does not contain \( \pi \) at all. The second is a divergence, which by Section 2.6 is non-negative and vanishes only when its two arguments are equal. So the objective is maximized exactly when \( \pi = \pi^* \), and no calculus was required — only the fact that a KL divergence is zero at its floor and positive everywhere else.
\( Z \) is intractable: it sums over all possible responses, and there are more of those than atoms in anything. That is why this form alone does not give an algorithm — and why what happens to \( Z \) in Move 3 is the whole of the discovery.
Move 2 — invert. Solve the same equation for the reward:
\[ r(x,y) = \beta \log\frac{\pi^*(y\mid x)}{\pi_{\text{ref}}(y\mid x)} + \beta\log Z(x). \]The reward is a log-ratio of the optimal policy to the reference, plus an intractable term that depends only on the prompt \( x \), not on the response \( y \).
Move 3 — substitute, and watch \( Z \) cancel. The preference model of Section 14.2 involves only the difference of two rewards for the same prompt. Substituting,
\[ r(x,y_w) - r(x,y_l) = \beta\log\frac{\pi^*(y_w)}{\pi_{\text{ref}}(y_w)} - \beta\log\frac{\pi^*(y_l)}{\pi_{\text{ref}}(y_l)} + \underbrace{\beta\log Z(x) - \beta\log Z(x)}_{=\ 0}. \]The intractable term cancels exactly, because it is the same for both responses. Putting this difference into the Bradley–Terry loss gives a loss over the policy alone:
\[ \mathcal{L} = -\log\sigma\!\left(\beta\log\frac{\pi(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta\log\frac{\pi(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\right). \]No reward model. No sampling loop. No value network. A supervised loss on preference pairs that optimizes the same KL-regularized objective — this is direct preference optimization, DPO, and the collapse above is the whole of it. The whole apparatus collapsed because an intractable constant was identical on both sides of a difference — the same cancellation-by-differencing that made Section 14.2's rewards identifiable only up to a constant.
The gradient has an instructive shape. Writing \( M \) for the margin inside the logistic, the loss is \( -\log\sigma(M) \) and its derivative carries a factor \( \sigma(-M) \). Pairs the model currently gets wrong — small or negative margin — receive weight near 1; pairs it already handles well receive vanishing weight. The method automatically concentrates on what it has not yet learned.
One step, by hand
Let \( \beta = 0.2 \), with log-ratios \( \log\frac{\pi(y_w)}{\pi_{\text{ref}}(y_w)} = 1.0 \) and \( \log\frac{\pi(y_l)}{\pi_{\text{ref}}(y_l)} = -0.5 \).
Margin: \( M = 0.2\,(1.0 - (-0.5)) = 0.3 \). Loss: \( -\ln\sigma(0.3) = -\ln 0.574 = \) 0.554 nats. Gradient weight: \( \sigma(-0.3) = 0.426 \).
Train further until the margin reaches \( 1.0 \): the weight falls to \( \sigma(-1) = 0.269 \). The pair now matters less — exactly the desired behavior.
A sanity check worth remembering. At initialization the policy is the reference, so both log-ratios are zero, the margin is zero, and the loss is \( -\ln\sigma(0) = \ln 2 = \) 0.693. Any implementation whose loss does not begin at \( \ln 2 \) has a bug. This is the acceptance criterion for Lab 6.
This derivation changed the estimator, not the objective. Both the clipped policy-gradient method and the collapsed loss optimize the same KL-regularized target; one does it by sampling and one by a closed-form substitution. Statements like “the direct method optimizes a different alignment goal” are false as stated. A second, subtler error: dropping \( Z \) too early. It does not vanish because it is small or ignorable — it vanishes because it is identical on both sides of a difference. That cancellation is the entire discovery, and a derivation that discards \( Z \) before taking the difference has skipped the content.
With \( \beta = 0.1 \) and log-ratios \( 0.8 \) and \( 0.2 \), give the margin, the loss in nats, and the gradient weight.
Margin \( = 0.1(0.8-0.2) = 0.06 \). Loss \( = -\ln\sigma(0.06) = -\ln 0.515 = 0.664 \) nats. Weight \( = \sigma(-0.06) = 0.485 \).
Policy-gradient training needs a baseline to reduce variance, conventionally supplied by a learned value network of roughly the policy's size. Chapter 8's arithmetic makes the cost concrete: carrying policy and critic together roughly doubles the memory bill. An alternative — group-relative policy optimization, GRPO — computes the baseline from data instead of a network: sample a group of \( G \) responses to the same prompt, and standardize their rewards within the group,
\[ \hat A_i = \frac{r_i - \bar r}{s_r}, \]where \( \bar r \) and \( s_r \) are the group's mean and standard deviation. The group supplies its own baseline; no critic is needed. This suits verifiable rewards especially well — mathematics and code, where a checker returns 1 or 0 — which is why it underpins modern reasoning training.
Group advantages, and a degenerate case
A group of five responses earns rewards \( (1,1,1,0,0) \). Mean \( \bar r = 0.6 \); variance \( = \frac{3(0.4)^2 + 2(0.6)^2}{5} = \frac{0.48+0.72}{5} = 0.24 \), so \( s_r = 0.49 \). Advantages: correct responses get \( 0.4/0.49 = \) +0.82, incorrect get \( -0.6/0.49 = \) −1.22.
Now the case that must be known: a group where all responses are correct, or all incorrect. Then every \( r_i \) equals the mean, every numerator is zero, and every advantage is zero — the group contributes no learning signal at all. This is the arithmetic reason such training curates prompts near the edge of the model's ability: problems it always solves and problems it never solves are both worthless.
A group of eight responses earns \( (1,0,0,0,0,0,0,0) \). Give all eight advantages.
Mean \( = 0.125 \); variance \( = \frac{(0.875)^2 + 7(0.125)^2}{8} = \frac{0.7656+0.1094}{8} = 0.1094 \), \( s_r = 0.331 \). Correct: \( 0.875/0.331 = +2.65 \). Each incorrect: \( -0.125/0.331 = -0.378 \).
Preferences become a reward through a logistic model of comparisons, identified only up to an additive constant. Optimization against that reward must be leashed to a reference policy, or it degenerates into exploiting the reward model's errors. The leashed objective has a closed-form optimum, and inverting it and substituting into the preference model cancels the intractable normalizer — collapsing the whole pipeline into a supervised loss whose gradient automatically weights the pairs still being gotten wrong. And replacing a learned baseline with a group's own statistics deletes the critic, halving the memory and fitting verifiable-reward tasks exactly. Chapter 15 takes up what happens when the model is asked to spend more compute at answer time rather than training time.
A · Drills
B · Problems
C · Challenge
Reproduce Derivation 14.2 — all three moves, with the cancellation shown explicitly — on blank paper, from nothing, on three separate days. Do not reread between attempts. Also reproduce the clip table (B-1) perfectly and the group-advantage computation including the degenerate case. This is the third and last protocol derivation, and it is the one most likely to be asked of you in a professional setting: the collapse is short, elegant, and widely misremembered.