RNN and LSTM

In “It is raining; remember your umbrella,” the final word depends on earlier information. An RNN updates a hidden state as it reads each position. An LSTM adds controls for keeping, writing, and reading its state. First we follow the information forward, then examine why learning long dependencies is difficult. An RNN squeezes all history into one fixed-size state and updates it step by step; LSTM gates make that state easier to keep, but the computation is still serial and the capacity still fixed.

How the hidden state carries information

An RNN updates a fixed-size hidden state rather than storing every token it has read. An LSTM maintains two vectors: a cell state for memory and a hidden state used for the current output and the next step's gates. Keeping these states separate makes the gate equations easier to follow.

The state update of a vanilla RNN

$$h_t = \tanh(W_x x_t + W_h h_{t-1} + b), \qquad y_t = W_y h_t$$

Each step receives only the current token vector $x_t$ and the previous state $h_{t-1}$. The parameters are shared across all time steps, which is why it can handle sequences of different lengths.

flowchart LR
    X1["x₁"] --> H1["h₁"]
    H1 --> H2["h₂"]
    X2["x₂"] --> H2
    H2 --> H3["h₃"]
    X3["x₃"] --> H3

“Recurrent” does not mean the graph really contains an infinite loop; it means the same cell is unrolled $T$ times along time.

Why early information gradually disappears

During training, gradients to earlier states pass through a sequence of Jacobians: the derivative matrices $\partial h_t/\partial h_{t-1}$ of each state with respect to the previous one. The weights are shared, but each Jacobian also depends on the current inputs, hidden state, and tanh derivative. Repeated contraction along a direction can shrink the gradient toward zero; repeated expansion can make it explode. Gating changes this propagation path rather than merely adding parameters.

So this is not just a matter of “a few more parameters would fix it.” The real trouble is that both information and gradients have to cross the same narrow state path over and over; the farther apart two positions are, the more easily something is lost on the way.

LSTM manages information with gates

An LSTM splits the state into a short-term output $h_t$ and a more direct memory path $c_t$:

$$f_t = \sigma(W_f[x_t;h_{t-1}] + b_f), \qquad i_t = \sigma(W_i[x_t;h_{t-1}] + b_i)$$

$$\tilde c_t = \tanh(W_c[x_t;h_{t-1}] + b_c), \qquad c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t$$

$$o_t = \sigma(W_o[x_t;h_{t-1}] + b_o), \qquad h_t = o_t \odot \tanh(c_t)$$

• forget gate $f_t$: how much of the old memory is kept;
• input gate $i_t$: how much of the new candidate is written;
• output gate $o_t$: how much of the current memory is exposed.

The important part is the additive path inside $c_t$. A forget gate near 1 causes less attenuation along this direct path. New writes can still change the state, and other gradient paths remain, so this does not guarantee successful long-range learning.

Trace the three gates with one number

Take one component of the cell state, initially 0.8. These gate values are chosen for arithmetic, not measured in a trained model.

A small output gate does not erase the cell state. A forget gate near 1 does not stop new writes. Substituting numbers helps distinguish what each gate actually controls.

What limits does an LSTM still have?

1. No parallelism over time: $h_t$ depends on $h_{t-1}$.
2. Single-state bottleneck: the information in a long sequence keeps being squeezed into a fixed-size vector.
3. Paths are too long: for the first token to influence the last one takes $T$ updates.

An LSTM mitigates forgetting, but it does not eliminate the sequential computation and the fixed-state bottleneck that come with recurrence.

Experiment: verify long-range dependencies

../code/sequence_numpy.py unrolls the RNN and LSTM forward computation in NumPy; ../code/sequence_torch.py has both learn a delayed-copy task and compares their error under long dependencies.

For input [3, 8, 2, 5, 1] and delay = 2, predictions from the third position onward target [3, 8, 2]: each target is two positions back, not the current input. From the repository root, run:

python 00-foundations/code/sequence_torch.py --task delay --model rnn --delay 8 --length 96
python 00-foundations/code/sequence_torch.py --task delay --model lstm --delay 8 --length 96

Then change delay to 64 while keeping length = 96; require 0 < delay < length. The script prints training-batch metrics, not validation results. Equal hidden widths also give these cells different parameter counts. Treat this as a way to inspect training behavior, not an architecture ranking from one run. A generalization comparison needs fixed evaluation data, parameter and training-budget accounting, and multiple seeds.

Self-check

Next

An RNN can read a sequence, but how do we turn an input sequence into an output sequence of a different length? Continue to Seq2Seq.

Quick learning: what do RNNs and LSTMs actually solve?