Neural Networks

RNN vs LSTM Memory

Feed in a sequence one token at a time and watch the hidden state fade in an RNN or get preserved via gates in an LSTM.

Comparing information flow in RNN and LSTM

Hidden State (h)
Cell State (c)
Forget Gate (f)

Final Hidden State (h)0.1719

An RNN is just a dense layer whose output loops back as an input to itself on the next step. Here is the loop before we unroll it over time.

Check your understanding

3 questions in the bank. Each attempt draws a fresh set in a fresh order, so a second go is a real second go.

The fundamental flaw of the RNN

A standard Recurrent Neural Network (RNN) processes sequences by maintaining a "hidden state" — a vector of numbers that represents its memory of everything it has seen so far. At each time step, it takes the current input and the previous hidden state, multiplies them by a set of weights, applies a squashing function like tanh, and outputs a new hidden state.

This works perfectly for short sequences. But look what happens when the sequence gets long.

Because the exact same weight matrix is applied at every single time step, the gradients during training are multiplied by that same matrix repeatedly. If the weights are even slightly less than 1, this repeated multiplication causes the gradients to shrink exponentially. This is the vanishing gradient problem. As a result, the network physically cannot learn to connect early inputs to late outputs. The memory "fades."

The LSTM solution: A separate memory cell

Long Short-Term Memory (LSTM) networks solve this by adding a second, separate memory channel called the Cell State ( $c$ ), alongside the hidden state ( $h$ ).

Instead of replacing the memory at every step with a completely new matrix multiplication, the LSTM updates the cell state using addition. Information can flow straight down the timeline unchanged, like a conveyor belt, unless the network explicitly decides to alter it. Because the update is additive, the derivative is 1, and the gradient flows back perfectly without vanishing.

The Gates

To control this conveyor belt, the LSTM uses three "gates," which are just small neural networks (sigmoid layers) that output numbers between 0 and 1.

Forget Gate: Decides what to throw away from the past. A value of 1 means "keep completely," 0 means "forget completely."
Input Gate: Decides which new information is worth adding to the cell state right now.
Output Gate: Decides what part of the cell state should be exposed as the hidden state for this specific time step.

When an LSTM fails

LSTMs only preserve memory if their forget gates remain open. Try moving the Forget Gate Bias slider into the negative.

A strongly negative bias forces the sigmoid function to output values close to 0, clamping the forget gate shut. When the forget gate is closed, the cell state is zeroed out at every step, and the additive information highway is severed. The LSTM degrades into acting exactly like a standard RNN, and the sequence is immediately forgotten. This is why it is standard practice to initialize the forget gate bias to a positive number (like 1.0) when training an LSTM.

Explore Next

Reference

RNN Update: h_t = tanh(W_h * h_{t-1} + W_x * x_t + b)
Vanishing Gradient: Repeated multiplication by values < 1 causes early gradients to shrink to zero.
Forget Gate (f): f_t = sigmoid(W_f * [h_{t-1}, x_t] + b_f). Decides what to throw away.
Input Gate (i): i_t = sigmoid(W_i * [h_{t-1}, x_t] + b_i). Decides what to update.
Cell State Update: c_t = f_t * c_{t-1} + i_t * g_t. Addition, not multiplication!
Output Gate (o): o_t = sigmoid(W_o * [h_{t-1}, x_t] + b_o). Decides what to output as h_t.

Break it on purpose

If the forget gate bias is strongly negative, the LSTM behaves exactly like an RNN: it closes the gate at every step, dropping the previous cell state and forgetting the sequence.