Vanilla RNN

Reuse the same small network at every time step, and let a hidden state carry forward whatever happened earlier so order in the sequence actually matters.

Unrolled across three time steps, an RNN is the same cell and the same weights reused at every step, with the hidden state carrying context forward

Why Does This Exist?

Feed a sentence into a multi-layer perceptron one word at a time and something is missing at once. The network has no memory. Show it bank after river and it responds exactly as bank after money. Nothing carries over. A fixed size feedforward net also wants a fixed size input, and sentences do not come in one length.

The real problem is order. Dog bites man and man bites dog share every word and mean opposite things. A bag of words cannot tell them apart, and many tasks hinge on that difference.

A recurrent network fixes both with one idea. Process one element at a time and keep a running summary, the hidden state, updated each step and carried into the next. The network never sees the whole sequence at once. It sees one token plus a summary of everything before it.

Think of It Like This

Reading a novel one page at a time with a running note

You do not hold every word of a 300 page novel in your head at once. You read page 40, update your sense of the plot, then read page 41 with that updated sense already in place. You do not reread pages 1 through 40 to catch up.

That running note is the hidden state. It does not store every word. It is a compressed sketch of what mattered so far. Read a twist on page 41 and the note updates. Forget a minor name from page 3 and it may be gone. The note has finite room, which is exactly the limit that motivates gated cells later.

Where it stops. You can flip back a few pages when confused. A vanilla RNN cannot reread. Its only past is the note it carried forward.

How It Actually Works

One cell reused every step

At step $t$ the cell takes the current element $x_t$ plus the previous hidden state $h_{t-1}$ and makes a new hidden state:

h_t = tanh(W_{xh} x_t + W_{hh} h_{t-1} + b)

Here $W_{xh}$ maps input to hidden space, $W_{hh}$ maps old hidden state to hidden space, $b$ is a bias, $tanh$ squashes to $(-1, 1)$ , $x_t$ is the current token vector, and $h_t$ is the running summary. The point visible above: $W_{xh}$ , $W_{hh}$ , and $b$ are the same three objects at every step. One cell applied $T$ times for length $T$ , not $T$ separate cells. That shared reuse is what recurrent means, and why any length works with the same weights.

An output at each step, for tagging each word say, comes from a readout $y_t = W_{hy} h_t$ , where $W_{hy}$ maps hidden state to output and $y_t$ is the prediction at position $t$ .

Worked example

Use two hidden numbers, $W_{xh} = [[0.5, -0.3], [0.2, 0.4]]$ , $W_{hh} = [[0.3, 0.1], [-0.2, 0.4]]$ , $b = [0.0, 0.0]$ , start $h_0 = [0.0, 0.0]$ , inputs $x_1 = [1.0, 0.0]$ , $x_2 = [0.0, 1.0]$ , $x_3 = [1.0, 1.0]$ . Then

h_1 = [0.462, 0.197], h_2 = [-0.141, 0.368], h_3 = [0.192, 0.650]

Each line calls the same function with the same matrices. Only $x_t$ and $h_{t-1}$ change. The first slot moves $0.462$ to $-0.141$ to $0.192$ as new inputs arrive, which is order mattering in numbers. The diagram above draws these three copies with one shared weight arc.

Training through time and where it strains

Because the same weights appear each step, training runs backpropagation through the unrolled graph. This variant is called backpropagation through time. Gradients flow backward through every hidden state, and the update for shared weights sums across every step that used them.

Chain many steps and the gradient becomes a long product of similar factors. Values below one shrink toward zero, values above one grow fast. In practice plain RNNs reliably lose signal from more than about 10 to 20 steps back. That finite memory is why long short-term memory adds gates that protect the carried note.

Code

One cell called three times, so shared weights stay visible in the loop.

1import math2
3
4def rnn_step(x: list[float], h: list[float], Wxh: list[list[float]], Whh: list[list[float]], b: list[float]) -> list[float]:5    return [math.tanh(sum(Wxh[i][j] * x[j] for j in range(2)) + sum(Whh[i][j] * h[j] for j in range(2)) + b[i]) for i in range(2)]6
7
8Wxh = [[0.5, -0.3], [0.2, 0.4]]; Whh = [[0.3, 0.1], [-0.2, 0.4]]; b = [0.0, 0.0]9h = [0.0, 0.0]10for t, x in enumerate([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]], start=1):11    h = rnn_step(x, h, Wxh, Whh, b)  # same matrices every call12    print(t, [round(v, 3) for v in h])13# -> 1 [0.462, 0.197]14# -> 2 [-0.141, 0.368]15# -> 3 [0.192, 0.65]

Three calls, untouched matrices, outputs matching the hand arithmetic above.

Watch Out For

Blaming data when gradients fade

Symptom: a plain RNN learns patterns inside a few tokens but never picks up context from far earlier, no matter how many epochs run or how wide the hidden state gets. Adding data repeats the same weak update.

Fix: check range directly with a plotted dependence longer than 20 steps. If short range works and long range never does, the cell is the limit. Move to a gated cell rather than more training.

Truncating the backward pass and forgetting it

Symptom: the forward pass runs 10,000 steps while the backward pass cuts off after 50 for speed. The model never learns a 200 step dependence, and the team tunes everything except the cutoff.

Fix: know the truncation window in use and never expect credit past it. Log the window beside the run, and lengthen it before blaming the data for a long miss.

The Quick Version

A vanilla RNN applies one cell with one weight set repeatedly, not one cell per position.
The hidden state is a fixed size running summary from current input plus previous hidden state.
The worked three steps give $[0.462, 0.197]$ , then $[-0.141, 0.368]$ , then $[0.192, 0.650]$ with identical weights.
Training uses backpropagation through time on the unrolled shared graph.
Long products shrink or grow, so plain cells reliably strain past 10 to 20 steps.