Core ML

Gated Recurrent Unit

Merge the LSTM's two states into one and cut three gates down to two, trading a little modeling flexibility for a noticeably smaller, faster-to-train cell.

A GRU merges the LSTM's cell state and hidden state into one line, and its update gate alone decides how much of the old state to keep versus overwrite with a new candidate

Why Does This Exist?

The LSTM solved the vanishing-gradient wall convincingly, but it did it with four separate weight matrices per layer — one each for the forget, input, and output gates, plus one for the candidate — and two pieces of state to carry between steps. That's a real cost: more parameters to learn, more matrix multiplies per step, and in 2014, training recurrent networks at any real depth or sequence length was expensive enough that the extra weight mattered.

The question worth asking is whether an LSTM's three gates are all doing genuinely separate jobs, or whether some of that machinery is redundant. It turns out two gates get most of the way there. A gated recurrent unit (GRU) answers that directly: same gating philosophy — control what's kept versus overwritten, rather than recomputing everything from scratch — with less machinery to train and less state to carry between steps.

Think of It Like This

One inbox instead of a notebook plus a filing cabinet

The LSTM's analogy was a whiteboard with an eraser, a marker, and a window onto it — three separate actions on one board, plus a filtered view of that board for anyone reading it. A GRU is closer to a single inbox with an autoresponder: for every new message, one decision covers it — keep the last summary exactly as it was, or replace it with something built from the new message. There's no separate "board" and "window onto the board." What's stored is what gets read, directly, with nothing filtered in between.

Fewer moving parts, one merged decision instead of three separate ones. Often close enough, and cheaper every time it runs.

How It Actually Works

One state, two gates

A GRU drops the LSTM's separate cell state entirely — there's only $h_t$ , playing both roles at once. The diagram above shows the two gates that control it.

The reset gate decides how much of the previous hidden state to use when building a new candidate:

r_t = \sigma(W_r [x_t, h_{t-1}] + b_r)

\tilde{h}_t = \tanh(W_h [x_t, r_t \odot h_{t-1}] + b_h)

When $r_t \approx 0$ in some dimension, the candidate is built as though the previous state in that dimension didn't exist — useful right at a boundary where the past genuinely stops being relevant (the start of a new clause, say).

The update gate then decides how much of the old state to keep versus overwrite with that candidate:

z_t = \sigma(W_z [x_t, h_{t-1}] + b_z)

h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t

This single line is doing the job the LSTM split across its forget and input gates. $z_t$ near 0 keeps $h_{t-1}$ almost entirely — the "remember" behavior a near-1 forget gate gave an LSTM. $z_t$ near 1 replaces the state with the fresh candidate — closer to an LSTM's input gate firing while its forget gate closes. One gate, both jobs, because $(1-z_t)$ and $z_t$ are forced to sum to exactly 1 rather than being learned independently.

What that coupling costs, and what it buys

An LSTM's forget and input gates are independent: it can, in principle, learn to forget a lot and write a lot in the same step, or forget nothing and write nothing, holding the state exactly still while still processing the input for some other purpose. A GRU's single update gate can't represent that — keeping and overwriting are strictly traded off against each other, because $z_t$ and $1-z_t$ move together by construction.

In practice, that constraint costs little on most sequence lengths and tasks — GRUs consistently land close to LSTMs in accuracy across language modeling and translation benchmarks, sequence for sequence. What it buys back is real: one state instead of two, and three weight matrices instead of four, which measurably speeds up training and shrinks the parameter count, particularly noticeable on smaller datasets where the LSTM's extra capacity was mostly unused anyway.

Show Me the Code

The GRU step, with the reset gate visibly acting before the candidate is built — the detail easiest to get backwards when implementing one by hand.

1import numpy as np2
3
4def sigmoid(z: np.ndarray) -> np.ndarray:5    return 1.0 / (1.0 + np.exp(-z))6
7
8def gru_step(x_t: np.ndarray, h_prev: np.ndarray, w: dict[str, np.ndarray]) -> np.ndarray:9    z_in = np.concatenate([x_t, h_prev])10    r_t = sigmoid(w["r"] @ z_in)                       # reset gate, computed first11    z_t = sigmoid(w["z"] @ z_in)                       # update gate12    gated_prev = r_t * h_prev                          # reset applied BEFORE the candidate13    h_tilde = np.tanh(w["h"] @ np.concatenate([x_t, gated_prev]))14    return (1 - z_t) * h_prev + z_t * h_tilde           # z_t and (1 - z_t) sum to exactly 115
16
17rng = np.random.default_rng(0)18weights = {k: rng.normal(scale=0.3, size=(4, 6)) for k in ("r", "z", "h")}19h = np.zeros(4)20h = gru_step(np.array([1.0, 0.0]), h, weights)21print(np.round(h, 3))  # -> [0.001 0.031 0.031 0.006]

Notice gated_prev uses r_t, not h_prev directly — the reset gate has already acted by the time the candidate sees the previous state. Get that order backwards and the reset gate does nothing at all.

Watch Out For

Treating GRU as strictly better because it's smaller

Fewer parameters and faster training don't mean a GRU dominates an LSTM. On long, information-dense sequences — long documents, or tasks where remembering and writing genuinely need to happen independently in the same step — the LSTM's extra gate can win by a real margin, because the GRU's coupled update gate can't represent "keep everything and also write something new" as a single-step choice. Benchmark both on your actual task before defaulting to the smaller cell; "usually close enough" is a statement about averages, not your specific sequence length and data.

Forgetting the reset gate exists and using update alone

Some hand-rolled or simplified GRU implementations drop the reset gate and use only the update gate, on the reasoning that update is "the important one." That's a different, weaker cell — without a reset gate, the candidate always incorporates the full previous state, so the network loses the ability to build a candidate that deliberately ignores stale context at a genuine sequence boundary. If you're implementing a GRU rather than using a framework's built-in layer, keep both gates.

The Quick Version

A GRU merges an LSTM's cell state and hidden state into one, and replaces three gates with two.
The reset gate controls how much of the previous state feeds into a new candidate; the update gate blends that candidate against the previous state.
Update and its complement sum to exactly 1, so keeping and overwriting are coupled — an LSTM's forget and input gates aren't.
Accuracy is usually close to an LSTM's; parameter count and training speed are consistently better.
Reset acts before the candidate is built — a common implementation bug is applying it in the wrong order or skipping it.