For two decades the obstacle to deep learning was not representation but optimization. The problem, and its cure, are a story about the shape of a valley.
Picture a narrow valley with steep walls. You are standing on one slope, blindfolded, allowed only to feel which way the ground falls and step that way. You will bounce from wall to wall and creep along the floor. That, almost exactly, is what training a deep network looked like for twenty years, and understanding why requires nothing more than the shape of the valley.
It is tempting to assume the difficulty is one of expressiveness — that among all possible settings of the weights, the rare good one is hard to find. For a long time the opposite was true. Good settings are plentiful. The trouble was that gradient descent, the only search procedure that scales, could not travel to them. It would take a step, overshoot in one direction, crawl in another, and stall. Naming that temptation is worth a sentence, because believing it sends you hunting for the wrong fix — more capacity, more regularization — when the problem was never capacity at all.
Near a minimum, any smooth loss looks like a bowl. Along some directions the walls are steep; along others they are shallow. If we align our coordinates with the natural axes of the bowl, the loss separates into independent one-dimensional pieces, each a simple parabola with its own steepness. A convenient model of a two-dimensional slice is
\[ L(x, y) = \tfrac{1}{2}\left(\lambda_1 x^2 + \lambda_2 y^2\right), \]where \( \lambda_1 \) and \( \lambda_2 \) are the steepnesses — the curvatures — along the two axes. If \( \lambda_1 \) is much larger than \( \lambda_2 \), the bowl is a long narrow trough: steep across, shallow along. The number that captures this asymmetry is the ratio of the largest curvature to the smallest,
\[ \kappa = \frac{\lambda_{\max}}{\lambda_{\min}}, \]called the condition number. A well-shaped bowl has \( \kappa \) near 1; a pathological valley has \( \kappa \) in the hundreds or thousands. Everything that follows is a statement about how badly a large \( \kappa \) hurts, and what can be done about it.
Gradient descent updates each coordinate by subtracting the step size \( \eta \) times the gradient. For our model surface the gradient along \( x \) is \( \lambda_1 x \), so the update is \( x \leftarrow x - \eta \lambda_1 x = (1 - \eta\lambda_1)x \), and likewise for \( y \). Each coordinate is simply multiplied, every step, by a fixed factor. From Chapter 1's stability reflex we know such an iteration converges exactly when the magnitude of the factor is below one.
Before deriving it, guess. If one direction of the bowl is a hundred times steeper than another, how much does that slow you down — a little, or a lot? Most people's instinct is that it costs you a constant factor, some fixed tax for an awkward shape. The instinct is wrong, and wrong in an important way: the cost is proportional to the ratio itself, so a hundredfold asymmetry costs you roughly a hundredfold in steps. Here is where that comes from.
The stable range of step sizes
The coordinate \( x \) obeys \( x \leftarrow (1 - \eta\lambda_1)x \), which converges to zero if and only if \( |1 - \eta\lambda_1| < 1 \), i.e. \( 0 < \eta\lambda_1 < 2 \). The same holds for \( y \) with \( \lambda_2 \). Both must hold at once, and since \( \lambda_1 = \lambda_{\max} \) is the larger curvature, it is the binding one:
\[ 0 < \eta < \frac{2}{\lambda_{\max}}. \]This is the whole trap in one line. The steep direction sets a ceiling on the step size. Any larger step and the iteration diverges along that axis, flinging the weights outward. So the step size is hostage to the steepest curvature in the entire problem — even though most directions would happily accept a far larger step.
Now watch what that ceiling does to the shallow direction. With \( \eta \) pinned just under \( 2/\lambda_{\max} \), the shallow coordinate contracts by a factor \( |1 - \eta\lambda_{\min}| \), which is close to \( 1 - 2\lambda_{\min}/\lambda_{\max} = 1 - 2/\kappa \). When \( \kappa \) is large this factor is barely below one: the shallow direction crawls. The steep direction, meanwhile, is oscillating back and forth across the valley, using up the entire step budget just to stay stable. The network is not stuck because it cannot represent the answer; it is stuck because the geometry forces it to inch down the valley floor while ricocheting between the walls.
We can make the slowness precise. The error along the shallow direction contracts by a fixed factor each step, so reaching a target reduction is a logarithm problem of exactly the kind Chapter 1 drilled. It is a standard result — we take it as given here, the way a physics problem gives you a formula for terminal velocity — that gradient descent on such a surface, at the best possible fixed step size, contracts the error each step by
\[ \rho_{\text{GD}} = \frac{\kappa - 1}{\kappa + 1}, \]while the same surface, optimized with a well-tuned momentum term that lets the shallow direction accumulate speed, contracts by
\[ \rho_{\text{mom}} = \frac{\sqrt{\kappa} - 1}{\sqrt{\kappa} + 1}. \]The number of steps to shrink the error by a factor \( F \) is \( t = \ln F / \ln(1/\rho) \). The two rates above look similar, but the square root inside the momentum rate is decisive, as the worked example shows.
Three hundred and forty-five steps, or thirty-four
Take a valley with \( \kappa = 100 \) — steep direction a hundred times sharper than the shallow one — and ask how many steps each method needs to reduce the error a thousandfold.
For plain gradient descent, \( \rho_{\text{GD}} = 99/101 = 0.9802 \), so \( t = \ln(1000)/\ln(1/0.9802) = 6.908/0.02 = 345 \) steps. For momentum, \( \rho_{\text{mom}} = 9/11 = 0.8182 \), so \( t = 6.908/\ln(1/0.8182) = 6.908/0.2007 = 34 \) steps. The ratio is almost exactly ten — which is \( \sqrt{\kappa} = \sqrt{100} \).
This is the quantitative content of the phrase “momentum glides along the valley floor.” Plain descent pays a cost proportional to \( \kappa \); momentum pays a cost proportional to \( \sqrt{\kappa} \). On a valley conditioned at \( 10^4 \), that is the difference between ten thousand steps and one hundred.
Why the speedup is the square root
The two rates differ only by a square root inside them, and it is worth seeing exactly how that becomes a square root in the step count. Expand each for large \( \kappa \).
For plain descent, \( \rho_{\text{GD}} = \frac{\kappa-1}{\kappa+1} \approx 1 - \frac{2}{\kappa} \). Since \( \ln(1-x) \approx -x \) for small \( x \),
\[ \ln\frac{1}{\rho_{\text{GD}}} \approx \frac{2}{\kappa}. \]For momentum, \( \rho_{\text{mom}} = \frac{\sqrt\kappa-1}{\sqrt\kappa+1} \approx 1 - \frac{2}{\sqrt\kappa} \), so \( \ln(1/\rho_{\text{mom}}) \approx 2/\sqrt\kappa \).
The step count to shrink the error by a factor \( F \) is \( t = \ln F/\ln(1/\rho) \), so the ratio of counts is the inverse ratio of those logarithms:
\[ \frac{t_{\text{GD}}}{t_{\text{mom}}} \approx \frac{2/\sqrt\kappa}{2/\kappa} = \sqrt{\kappa}. \]The whole effect is that momentum's rate carries \( \sqrt\kappa \) where plain descent carries \( \kappa \), and the logarithm turns a difference of rates into a ratio of counts. At \( \kappa = 100 \) that is the factor of ten of Worked Example 3.1; at \( \kappa = 10^4 \) it is a hundred.
A valley has \( \kappa = 400 \). Estimate the ratio of plain-gradient-descent steps to momentum steps for the same error reduction, without computing either count in full.
The ratio of step counts is \( \ln(1/\rho_{\text{GD}}) \) versus \( \ln(1/\rho_{\text{mom}}) \) inverted, which for large \( \kappa \) tends to \( \sqrt{\kappa} \). Here \( \sqrt{400} = 20 \). (Explicitly: \( \rho_{\text{GD}} = 399/401 \Rightarrow \ln(1/\rho) = 0.005 \); \( \rho_{\text{mom}} = 19/21 \Rightarrow \ln(1/\rho) = 0.100 \); ratio \( 0.100/0.005 = 20 \).)
The deeper failure this chapter describes is a training failure, not overfitting. When a plain deep network does worse than a shallow one, the instinct is to suspect it has memorized and failed to generalize. But the error that rises is the training error itself: the optimizer never reached a good setting at all. Confusing an optimization failure with a generalization failure sends you reaching for the wrong remedy — regularization instead of better conditioning — and it is one of the most common misreadings of the early deep-learning literature.
Conditioning governs how the optimizer moves once it starts; initialization governs whether it can start at all. Two separate requirements fall directly out of Chapter 1's variance algebra, and both were, historically, hard-won.
The first is scale. Recall the fan-in derivation: a unit summing \( m \) unit-variance inputs with weights of variance \( \sigma^2 \) produces an output of variance \( m\sigma^2 \). If \( \sigma^2 \) is chosen carelessly, signal either explodes or vanishes as it crosses each layer, and a deep stack multiplies that error many times over. Preserving the scale of activations from layer to layer requires
\[ \sigma^2 = \frac{1}{m}, \]the principle behind every sensible initialization scheme. The precise constant is adjusted for the nonlinearity in use, but the \( 1/m \) skeleton is always there.
A too-small initialization
A layer has fan-in \( m = 4096 \). The variance-preserving choice is \( \sigma^2 = 1/4096 \), i.e. \( \sigma \approx 0.0156 \). Suppose instead someone initializes with a flat \( \sigma = 0.01 \). The output variance is then \( m\sigma^2 = 4096 \times 10^{-4} = 0.41 \). Each layer shrinks the signal to under half its variance; across a dozen layers the activations have all but vanished, and with them the gradients. The network trains glacially or not at all — not because anything is conceptually wrong, but because a single variance was set a factor of \( 1.5 \) too small and compounded.
The second requirement is asymmetry. Suppose every weight in a layer were initialized to the same constant. Then every unit in that layer computes the identical function of the inputs, receives the identical gradient, and updates identically — forever. The layer, however wide, collapses to a single effective unit, and no amount of training breaks the tie. This is why initialization must be random: randomness is what breaks the symmetry between units and lets them specialize. It is a one-line argument, but it is the reason a fundamental-looking operation — “set the weights to a sensible constant” — is quietly fatal.
Symmetry never breaks itself
Consider two units in a layer, with incoming weight vectors \( w_1 \) and \( w_2 \), fed the same inputs. If \( w_1 = w_2 \) at initialization, then their outputs are equal at every input, so the loss depends on them only through their sum. The gradient of the loss with respect to \( w_1 \) therefore equals the gradient with respect to \( w_2 \), and one gradient step keeps \( w_1 = w_2 \). By induction the two are equal for all time. The only way to give them distinct gradients is to start them at distinct values — hence random initialization.
A small practical corollary closes the chapter, and it too is pure Chapter-1 reasoning. Units with a rectified activation — zero for negative inputs, linear for positive — contribute no gradient when their input is negative, and a unit that begins with a negative bias may sit dead from the start. Nudging the initial bias slightly positive shifts the input distribution rightward, so more units begin in their active region and receive gradient. The same variance-and-mean bookkeeping that governs the weights governs this choice; it is not a trick so much as an accounting consequence.
A layer has fan-in \( m = 1024 \). (a) What \( \sigma \) preserves unit output variance? (b) If instead \( \sigma = 0.05 \) is used, what is the output variance, and does the signal grow or shrink across the layer?
(a) \( \sigma = 1/\sqrt{1024} = 1/32 \approx 0.031 \). (b) Output variance \( = 1024 \times 0.0025 = 2.56 \); since this exceeds 1, the signal grows across the layer, and would explode across a deep stack.
Everything that follows rests on what this chapter established, and all of it came from Chapter 1's tools. The stable step size is capped by the steepest curvature, so a badly conditioned problem is slow no matter how patient you are. Momentum converts that \( \kappa \)-sized penalty into a \( \sqrt{\kappa} \)-sized one — the first appearance of a theme, adaptivity to geometry, that runs through the rest of the field. And initialization is a variance-and-symmetry problem whose answers are forced, not chosen. Chapter 4 takes up the complementary question: if optimization is this sensitive to the shape of the surface, can we build networks whose surfaces are easier to descend? The answer — residual connections — is where architecture begins to do the optimizer's job for it.
A · Drills
B · Problems
C · Challenge
Reproduce Derivations 3.1 (the stable step range) and 3.2 (the \( \sqrt{\kappa} \) speedup) on blank paper, then attempt B-1 closed book. This is the first of the three derivations carrying the three-separate-days protocol: return to the \( \kappa \)-versus-\( \sqrt{\kappa} \) result on two later days and reproduce it cold each time. It is the template for the compute-optimal derivation of Chapter 7, and the fluency you build here is the fluency you will spend there.