Gated Recurrent Unit

A recurrent cell that keeps old memory unless new input earns a rewrite. Two small gates decide what to forget when building a candidate and how much of it to keep.

Reset 0.5 and update 0.5 turn candidate 0.0 and state 0.8 into new state 0.40 on one shared memory line

Why Does This Exist?

A vanilla RNN rewrites its whole state every step, so old clues wash out fast. Long short-term memory fixes that with a separate cell plus three gates, at the cost of four weight matrices and two states to carry.

Many tasks do not need that full machinery. A gated recurrent unit keeps the gating idea with one state and two gates: a reset gate that decides how much past to use for a fresh candidate, and an update gate that blends that candidate against the old state. Three matrices instead of four, one line of memory instead of two, and training that runs faster.

This page covers the two gates and their tradeoff. The three gate parent lives in long short-term memory. The ungated starting point lives in vanilla RNN. Reading both directions lives in bidirectional RNN.

Think of It Like This

One notebook with two sliders

You keep trip notes in one notebook. Each new stop brings two sliders. The first slider sets how much of the old notes you reread before drafting a replacement paragraph. Slide it to zero at a new city and you draft from the fresh stop alone. Slide it to full mid city and you draft with full history beside you.

The second slider sets how much of the draft replaces the notebook. Slide it to zero and the notebook stays exactly as it was. Slide it to full and the draft overwrites it. Middle values blend.

Old notes are $h_{t-1}$ , the first slider is the reset gate $r_t$ , the draft is the candidate, the second slider is the update gate $z_t$ , and the notebook after is $h_t$ .

Where it stops: you can keep everything and still paste something new on the next page. The unit cannot, because keep and overwrite shares sum to one by construction.

How It Actually Works

Call $x_t$ the current input and $h_{t-1}$ the old state. Call $\sigma$ the sigmoid that squashes any number into $0$ to $1$ . Call $\odot$ elementwise multiply. Call $W_r$ , $W_z$ , $W_h$ the three weight matrices with biases folded in.

Reset gate first, since the candidate needs it:

r_t = \sigma(W_r [x_t, h_{t-1}])

Candidate built with reset applied to the past:

\tilde{h}_t = \tanh(W_h [x_t, r_t \odot h_{t-1}])

Update gate, then the blend that is the whole trick:

z_t = \sigma(W_z [x_t, h_{t-1}]), \quad h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t

When $r_t$ nears $0$ the candidate ignores the past, which helps at true boundaries such as a new sentence. When $z_t$ nears $0$ the state freezes. When $z_t$ nears $1$ the candidate overwrites. Keeping and overwriting trade off because $(1 - z_t)$ and $z_t$ sum to one, unlike the independent forget and input gates in long short-term memory.

Worked example: 0.8 carried to 0.40 in one step

Take scalar state $h_{t-1} = 0.8$ . Suppose the reset input sums to $0$ , so $r_t = \sigma(0) = 0.5$ . Suppose the candidate input sums to $0$ , so $\tilde{h}_t = \tanh(0) = 0.0$ . Suppose the update input sums to $0$ , so $z_t = \sigma(0) = 0.5$ .

Blend:

h_t = (1 - 0.5) \times 0.8 + 0.5 \times 0.0 = 0.40

That $0.8$ to $0.40$ move is the diagram headline: one horizontal memory line, a reset box feeding the candidate box from below, and a blend circle where $0.5$ keeps half the past.

What the coupling costs and buys

An LSTM can forget nothing and write a lot in the same step because its gates move independently. A GRU cannot pick that corner: more keep means less write by definition. On most mid length tasks the gap is small. What returns is real: one state to store, three matrices to learn, fewer multiplies per step, and less overfitting on small data where the fourth matrix mostly fit noise.

Code

Rebuild the worked example with plain Python and confirm $0.40$ .

1import math2
3def sigmoid(a: float) -> float:4    return 1.0 / (1.0 + math.exp(-a))5
6
7def gru_scalar(x: float, h_prev: float, pre_r: float, pre_z: float, pre_h: float) -> float:8    r: float = sigmoid(pre_r)9    z: float = sigmoid(pre_z)10    cand: float = math.tanh(pre_h + 0.0 * x + 0.0 * (r * h_prev))11    h_new: float = (1.0 - z) * h_prev + z * cand12    return h_new13
14
15h_prev: float = 0.816h_new: float = gru_scalar(1.0, h_prev, 0.0, 0.0, 0.0)17
18print(f"{sigmoid(0.0):.1f} {math.tanh(0.0):.1f}")  # -> 0.5 0.019print(f"{h_new:.2f}")  # -> 0.40

The printout matches the hand arithmetic. Set the reset pre value very negative so $r$ nears $0$ and watch the candidate drop its $h_{t-1}$ term entirely.

Watch Out For

Reading smaller as strictly better than LSTM

Fewer matrices train faster, so teams default to GRU everywhere. On long dense sequences where keep and write must fire together, the coupled gate loses real accuracy.

Symptom: GRU trails LSTM by a stable margin on the longest inputs while matching elsewhere. Fix: benchmark both on full length data from long short-term memory before locking the cell.

Applying reset after the candidate is built

The reset product $r_t \odot h_{t-1}$ must enter the tanh. Placed after, it scales the finished candidate and the boundary behavior vanishes.

Symptom: boundaries bleed and shortening the reset path changes nothing. Fix: follow the order in the code above and test with $r$ at $0$ : the candidate must ignore $h_{t-1}$ , as in vanilla RNN debugging.

The Quick Version

A GRU carries one state line instead of the two lines in an LSTM.
Reset 0.5 gates old state 0.8 before the candidate is built in our example.
Candidate 0.0 blends against old state 0.8 at update 0.5 to give new state 0.40.
Update and its complement sum to one, so keeping more always means writing less.
Three weight matrices replace four, which speeds training and shrinks the cell.
Reset must act before the candidate tanh, or boundary resets stop working.