A recurrent network remembers by multiplying. Whatever is multiplied a thousand times either vanishes or explodes — unless something pins the factor to one. That pin is the whole story.
A recurrent network remembers by multiplying. To carry information from step one to step three hundred, it must pass that information through three hundred multiplications, and whatever survives is what the network can recall. Multiply anything by 0.95 three hundred times and you have nothing left. Multiply by 1.0 and you have everything. Almost everything in this chapter follows from that observation.
Learning long-range structure requires a gradient to travel backward across many steps, and backward travel across a recurrence is a product of per-step factors. Chapter 4 showed what long products do: they die or explode unless something holds them in check. Here the same problem reappears along the time axis rather than the depth axis, and the fix has the same character. What follows derives the horizon over which memory survives, the single constant that sets it, and a consequence for regularization that catches most people out.
First the object itself, because the whole chapter turns on one of its terms. A gated cell carries a memory vector \( c_t \) from step to step. At each step it decides how much of that memory to keep and how much new material to add:
\[ c_t = f_t \odot c_{t-1} + i_t \odot g_t, \]where \( g_t \) is a candidate update computed from the input and the previous state, \( i_t \) is an input gate deciding how much of it to admit, \( f_t \) is the forget gate deciding how much of the old memory to retain, and \( \odot \) multiplies entrywise. Both gates are values in \( (0,1) \) produced by a logistic function — Section 2.2 — of a learned pre-activation, so the cell computes them afresh at every step rather than carrying them as fixed weights.
Now differentiate. Holding the gates fixed, \( \partial c_t/\partial c_{t-1} = \operatorname{diag}(f_t) \): the memory's sensitivity to its own past is the forget gate and nothing else. That is the entire reason this architecture exists, and it is why the factor in what follows is not an abstraction but a number the network chooses.
Backpropagation through time multiplies the gradient, at each step, by exactly that factor. Call it \( f \). After \( t \) steps the gradient carries \( f^t \). If \( f < 1 \), the influence decays geometrically; the number of steps over which a gradient survives is a logarithm problem of exactly the kind Chapter 1 drilled.
How far back memory reaches
A gradient scaled by \( f \) per step falls below a fraction \( \epsilon \) of its original size after \( t \) steps, where \( f^t < \epsilon \), i.e.
\[ t > \frac{\ln \epsilon}{\ln f}. \]Both logarithms are negative for \( f < 1 \) and \( \epsilon < 1 \), so \( t \) is positive. With \( f = 0.95 \) and \( \epsilon = 0.01 \): \( t > \ln(0.01)/\ln(0.95) = (-4.605)/(-0.0513) = 89.8 \), so the gradient is gone after about 90 steps. With \( f = 0.5 \), the horizon collapses to \( \ln(0.01)/\ln(0.5) \approx 6.6 \) steps — a network that cannot see past the last handful of tokens.
The critical case is \( f = 1 \). Then \( f^t = 1 \) for all \( t \): the gradient neither decays nor grows, and memory reaches arbitrarily far back. A recurrent cell that could hold its per-step factor at exactly one would have, in principle, unbounded memory. This is the target the gated cell was built to hit.
The device that hits it is the forget gate. The gated cell maintains a memory that is multiplied each step not by a fixed weight but by a gate value \( f \in (0,1) \) that the network computes. When the network wants to remember, it drives the gate toward one, and the memory is carried forward almost untouched — the “constant-error carousel” that gives the gradient a clear channel across time. When it wants to forget, it drives the gate down. The horizon is no longer a fixed property of the architecture; it is something the network sets, token by token.
A gate is produced by a squashing function of a learned pre-activation: \( f = \sigma(z) \), where \( \sigma \) is the logistic function and \( z \) includes a bias term. At the start of training, before the input-dependent part has learned anything, the gate sits at \( \sigma(b) \) for bias \( b \). If we want the cell to default to remembering — a sensible prior for tasks with long dependencies — we should choose \( b \) so that \( \sigma(b) \) is close to one.
The bias for a chosen default memory
To make the forget gate open to a value \( f \) at initialization, invert the logistic: \( f = \sigma(b) = \frac{1}{1+e^{-b}} \) gives
\[ b = \ln\frac{f}{1-f}. \]For \( f = 0.95 \): \( b = \ln(0.95/0.05) = \ln 19 = 2.94 \). Initializing the forget-gate bias near 3, rather than 0, starts the network with a memory horizon of about 90 steps (by Derivation 5.1) instead of the roughly 7 steps that \( b = 0 \), \( f = 0.5 \) would give. A single constant, chosen by a one-line inversion, changes the default reach of the network by an order of magnitude — which is why high forget-gate bias is standard practice.
(a) What forget-gate value gives a horizon (to the 1% level) of about 300 steps? (b) What bias produces it?
(a) Need \( \ln(0.01)/\ln f = 300 \Rightarrow \ln f = -4.605/300 = -0.01535 \Rightarrow f = 0.9848 \). (b) \( b = \ln(0.9848/0.0152) = \ln(64.8) = 4.17 \).
This next result catches almost everyone, including people who have used these networks for years. Dropout regularizes a network by randomly zeroing a fraction of its signals during training. On a feedforward network this is benign. On a recurrent network, where you apply it decides whether the network can remember at all — and the reason is once again the compounding of a per-step factor.
Noise on the recurrent path compounds in time
Suppose dropout with keep-probability \( 0.9 \) were applied to the memory as it is carried from each step to the next — on the recurrent path itself. Then over a sequence of 100 steps, the probability that a particular piece of state survives untouched is \( 0.9^{100} \). By Chapter 1's logarithm reflex, \( 0.9^{100} = e^{100\ln 0.9} = e^{-10.54} = 2.7\times 10^{-5} \). The carried memory is annihilated; the very pathway the forget gate worked to protect is destroyed by noise applied a hundred times over.
Now suppose instead the dropout is applied only to the vertical connections — the ones feeding from layer to layer at a single time step, not from step to step. Then a given piece of information is perturbed a number of times equal to the network's depth, perhaps two or three, regardless of how long the sequence is. The regularization does its job on the transformations between layers while leaving the memory-carrying pathway deterministic. That contrast — noise that compounds per time step versus noise that compounds per layer — is the entire content of a well-known result on recurrent regularization.
It is tempting to think inverted dropout's rescaling — dividing surviving signals by the keep-probability to preserve the mean — makes recurrent-path dropout harmless. It does not. Rescaling fixes the expectation, but multiplicative noise compounds in variance, and it is the variance that destroys the carried state over a long sequence. The design goal is not an unbiased mean; it is a memory pathway that stays deterministic.
Before we leave recurrence, a piece of bookkeeping we will reuse. A gated cell computes four quantities per step — a candidate update and three gates — each a linear map of the concatenated input and previous hidden state, plus a bias. For input width \( d_{\text{in}} \) and hidden width \( d_h \), each of the four maps has \( d_h(d_{\text{in}} + d_h) \) weights and \( d_h \) biases, so the layer holds
\[ 4\,d_h\,(d_{\text{in}} + d_h + 1) \]parameters. Assemble a full model from these and a pattern emerges worth noticing.
Where the parameters actually are
Take a language model with an embedding of width 1500 over a vocabulary of 10,000; two stacked recurrent layers each of hidden width 1500 (the first fed by the embedding); and an untied output projection back to the vocabulary, with a bias. Count each part.
Embedding: \( 10{,}000 \times 1500 = 15{,}000{,}000 \). First recurrent layer: \( 4 \times 1500 \times (1500 + 1500 + 1) = 6000 \times 3001 = 18{,}006{,}000 \). Second layer: identical, \( 18{,}006{,}000 \). Output projection: \( 1500 \times 10{,}000 + 10{,}000 = 15{,}010{,}000 \). Total: \( 66{,}022{,}000 \), about 66 million.
Notice that the embedding and output projection together are \( 30 \) million — about \( 45\% \) of the model — yet neither adds any depth; they are lookup and readout. This is precisely why scaling studies, when they relate capability to size, count non-embedding parameters: the vocabulary tables scale with vocabulary, not with the model's actual computational depth.
A single recurrent layer has input width 512 and hidden width 512. How many parameters does it hold?
\( 4 \times 512 \times (512 + 512 + 1) = 2048 \times 1025 = 2{,}099{,}200 \), about 2.1 million.
Recurrent models were where the field first watched error fall as a clean power law in the quantity of training data — and where the brutal economics of that law became visible. The reflex from Chapter 1, extracting an exponent from a “per tenfold” statement, is all we need to reason about it.
The price of halving an error
Suppose a speech system's word error rate obeys a power law in training hours, and each tenfold increase in data yields a \( 40\% \) relative reduction in error. Then error \( \propto D^{\beta} \) with \( 10^{\beta} = 0.6 \), so \( \beta = \log_{10} 0.6 = -0.222 \). To halve the error we need \( 10^{\beta x} = 0.5 \), i.e. \( x = \log_{10}(0.5)/0.222 = 0.301/0.222 = 1.36 \) decades of data — a factor of \( 10^{1.36} = 22.7 \).
Starting from a \( 10\% \) error at 12,000 hours, reaching \( 5\% \) therefore demands about \( 12{,}000 \times 22.7 \approx 273{,}000 \) hours. Each halving of error costs a roughly twenty-three-fold multiplication of data. Power laws are smooth and dependable, but they are not cheap — a theme Chapter 7 will make the center of the story.
A different system improves \( 30\% \) relative per tenfold data increase. What data multiple halves its error?
\( 10^\beta = 0.7 \Rightarrow \beta = \log_{10}0.7 = -0.155 \). Halving: \( x = 0.301/0.155 = 1.94 \) decades \( = 10^{1.94} \approx 88\times \).
Memory in a recurrent network is a per-step factor held near one; the horizon over which it survives is a logarithm, and the gate's initial bias sets that horizon by a one-line inversion. Regularization must respect the distinction between noise that compounds in time and noise that compounds in depth — a distinction that decides whether the network can remember at all. And error falls as a power law in data whose exponent you can read straight off a “per tenfold” statement, with consequences for cost that scale is about to make unavoidable. Chapter 6 introduces the mechanism that replaced the recurrent product entirely: attention, which reaches any past token in a single step, and whose own design choices — the \( \sqrt{d} \) scaling chief among them — fall straight out of the variance algebra we have been using all along.
A · Drills
B · Problems
C · Challenge
Reproduce Derivations 4.1 (the horizon), 4.2 (the bias inversion), and 4.3 (the compounding contrast) on blank paper. Attempt B-1 and B-2 closed book. You pass when the horizon logarithm and the per-step-versus-per-layer distinction are both automatic — the second is the one students most often get wrong under time pressure.