Long Short-Term Memory

Give the network a separate memory line that gates control instead of overwrite, so relevant information from many steps back can survive untouched until it's needed.

Cell moves from 1.0 to 1.38 by keeping 0.9 and adding 0.48 while the output gate reads 0.62

Why Does This Exist?

A vanilla RNN recomputes its hidden state fully at each step through a tanh. The gradient for a clue fifty steps back must survive fifty multiplies. In practice it shrinks toward zero. A subject twenty words back cannot steer its verb.

The 1997 fix did not tune that update. It added a second state whose job is to avoid the squash, edited by choice rather than overwritten. This page covers that memory line and its three gates. The plain starting point lives in vanilla RNN. The slimmer cousin lives in gated recurrent unit. Stacking such cells for translation lives in sequence to sequence.

Think of It Like This

A whiteboard with eraser, marker, and window

A project whiteboard holds notes for weeks. It is never wiped clean each meeting. Three small acts happen. Someone erases lines no longer needed, which is forget. Someone adds notes for what just came up, which is input. Someone glances and decides what to say aloud now, which is output, usually a filtered slice rather than the whole board.

The board is the cell state. It persists except where an act touches it. That differs from a vanilla RNN, where the whole board is rewritten every step whether anything changed or not.

Where it stops: meetings allow talk and erasure in any order with judgment. Gates are fixed multiplies between zero and one learned from data, with no notion of truth, and the window never changes what the board holds.

How It Actually Works

Three gates edit one memory line. Each gate is a sigmoid over current input $x_t$ plus prior hidden $h_{t-1}$ , giving per dimension numbers in $[0, 1]$ that mean how much.

Forget decides what stays:

f_t = \sigma(W_f [x_t, h_{t-1}] + b_f)

Near $0$ erases that dimension. Near $1$ keeps it. It scales old memory only and adds nothing.

Input decides what to add, with candidate $\tilde{c}_t = \tanh(W_c[x_t, h_{t-1}] + b_c)$ :

i_t = \sigma(W_i [x_t, h_{t-1}] + b_i)

Cell update combines both with elementwise multiply $\odot$ :

c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t

If $f_t \approx 1$ and $i_t \approx 0$ , then $c_t \approx c_{t-1}$ . Memory rides through untouched.

Output decides what others see:

o_t = \sigma(W_o [x_t, h_{t-1}] + b_o), \qquad h_t = o_t \odot \tanh(c_t)

Here $h_t$ feeds the next layer. It is a filtered view, not the memory itself.

Worked example with one step

Start cell $c_{t-1} = 1.0$ . Forget keeps $f_t = 0.9$ , so $0.9 \times 1.0 = 0.90$ . Input opens $i_t = 0.6$ on candidate $\tilde{c}_t = 0.8$ , so $0.6 \times 0.8 = 0.48$ . New cell is $0.90 + 0.48 = 1.38$ .

Squash with tanh. $\tanh(1.38) \approx 0.88$ . Output opens $o_t = 0.7$ , so $h_t = 0.7 \times 0.88 \approx 0.62$ . Memory moved $1.0$ to $1.38$ with a paper trail, and the next layer sees only $0.62$ . That move is the diagram headline.

Why gradients survive

Path from $c_{t-1}$ to $c_t$ multiplies by learned $f_t$ , not by a matrix through tanh. Learn $f_t \approx 1$ for a dimension worth keeping and the gradient through that step is near $1$ . Chain one hundred such steps and it still survives, because no saturating nonlinearity sat on the highway. Forget is both memory control and gradient control, which is the long in the name.

Code

One step in plain Python matching the hand numbers.

1import math2
3c_prev: float = 1.04f: float = 0.95i: float = 0.66c_tilde: float = 0.87o: float = 0.78
9c: float = f * c_prev + i * c_tilde10h: float = o * math.tanh(c)11
12print(f"{c:.2f} {math.tanh(c):.2f} {h:.2f}")  # -> 1.38 0.88 0.62

The printout matches $1.38$ , $0.88$ , $0.62$ . Set $f$ to $1.0$ and $i$ to $0.0$ and $c$ equals $c_{prev}$ exactly. Untouched memory by construction.

Watch Out For

Forget bias starts at erase by default

Near zero init gives $f_t = \sigma(0) = 0.5$ . Cell halves each step before learning anything.

Symptom: long range tasks learn slowly or stall while short ones look fine. Fix: init forget bias near $1.0$ so training starts at remember by default, and check your framework default before assuming neutral.

Feeding cell where hidden belongs

Cell $c_t$ is long memory. Hidden $h_t$ is the filtered output for this step. Same shape, different job.

Symptom: hand wired model trains worse with no crash, because output filtering never happens. Fix: pass $h_t$ onward and across time, keep $c_t$ on its highway, and assert shapes plus gate order in a tiny one step test like the code above.

The Quick Version

LSTM adds a cell state beside the hidden state, edited by three sigmoid gates.
Our step keeps 0.9 of 1.0 and adds 0.48 for a new cell of 1.38.
Tanh of 1.38 is about 0.88 and output gate 0.7 gives hidden about 0.62.
Forget near 1 preserves memory and gradient together across many steps.
Cell and hidden are different objects with different jobs and are not interchangeable.