Bidirectional RNNs

Run one chain forward and a second chain backward over the same sequence, then join both states at each position so every token sees what came before and after.

Forward states 0.66, 0.32 and 0.74 meet mirrored backward states over inputs 1.0, 0.0 and 1.0, concatenated at each position

Why Does This Exist?

Take the sentence "The bank raised its rates" and ask what "bank" means the moment you reach it. A plain vanilla RNN has seen only "The" so far, so a riverbank and a financial institution look identical. The word that settles it, "rates", sits three tokens in the future.

That gap is structural. A one-direction hidden state at position $t$ depends only on inputs $x_1$ through $x_t$ . When you generate text that limit is honest because the future does not exist yet. When the full sentence already sits in memory, it throws away help that is free.

This page covers the two-pass fix: one chain left to right, one right to left, joined at each step, with one hand-computed example. For the one-direction baseline start with vanilla RNN. For the gated cells people usually plug into both directions read Long Short-Term Memory and Gated Recurrent Unit. For the encoder that often stacks this idea see Sequence-to-Sequence.

Think of It Like This

Two surveyors, one street

You need the height of every house on a street. Surveyor A starts at the west end and walks east, writing at each house what she has passed so far. Surveyor B starts at the east end and walks west along the same street, writing at each house what lies ahead from the other side. At each door you staple both notes together, so the second house carries what stands on both sides of it.

The street is the sequence, each walk is one recurrent pass, and each stapled pair is the concatenated hidden state. Neither surveyor radios the other mid-walk, which matches the two independent weight sets.

Where it stops: both surveyors need the finished street before they start. You cannot survey a street that is still being built, which is why this trick fits labeling complete input and not live generation.

How It Actually Works

Write $x_t$ for the input at position $t$ , meaning the embedding of token $t$ . Write $\overrightarrow{h}_t$ for the forward hidden state at $t$ , meaning the summary built left to right, and $\overleftarrow{h}_t$ for the backward hidden state at $t$ , meaning the summary built right to left. Each direction has its own weights $W_f$ and $W_b$ plus biases $b_f$ and $b_b$ , and both use the same nonlinearity $f$ , usually $\tanh$ . Write $T$ for the sequence length and $h_t$ for the joined state you hand downstream.

The two passes

The forward chain starts from $\overrightarrow{h}_0 = 0$ and steps left to right:

\overrightarrow{h}_t = f(W_f [x_t ; \overrightarrow{h}_{t-1}] + b_f)

The bracket $[;]$ means concatenation of the current input with the previous hidden state. The backward chain starts from $\overleftarrow{h}_{T+1} = 0$ and steps right to left over the same inputs with its own weights:

\overleftarrow{h}_t = f(W_b [x_t ; \overleftarrow{h}_{t+1}] + b_b)

The two chains never read each other mid-pass, so you can run them in either order or side by side.

Joining the views

At each position staple both summaries together:

h_t = [\overrightarrow{h}_t ; \overleftarrow{h}_t]

A tagger or classifier reads $h_t$ , never one side alone. That doubled width is real: if each direction holds 64 numbers, $h_t$ holds 128, and the next layer must accept 128.

Worked example: three numbers through both directions

Keep it tiny so you can check by hand. Inputs $xs = [1.0, 0.0, 1.0]$ , one weight on the input $w_x = 0.8$ , one weight on the carried state $w_h = 0.5$ , activation $\tanh$ , start state $0$ . The update is $h = \tanh(w_x x + w_h h_{prev})$ , with full precision carried forward and rounded only for display.

Forward:

Step 1: $0.8 \times 1.0 + 0.5 \times 0 = 0.80$ , and $\tanh(0.80) \approx 0.66$ .
Step 2: $0.8 \times 0.0 + 0.5 \times 0.6640 \approx 0.3320$ , and $\tanh(0.3320) \approx 0.32$ .
Step 3: $0.8 \times 1.0 + 0.5 \times 0.3203 \approx 0.9602$ , and $\tanh(0.9602) \approx 0.74$ .

So forward states are $[0.66, 0.32, 0.74]$ .

Backward runs the same rule from the right, starting from $0$ past position 3:

Step 3: $0.80$ gives $0.66$ .
Step 2: $0.3320$ gives $0.32$ .
Step 1: $0.9602$ gives $0.74$ .

So backward states, realigned left to right, are $[0.74, 0.32, 0.66]$ .

Concatenate per position:

$h_1 = [0.66 ; 0.74]$
$h_2 = [0.32 ; 0.32]$
$h_3 = [0.74 ; 0.66]$

Position 2 is the payoff. Its forward half $0.32$ saw $1.0$ then $0.0$ , and its backward half $0.32$ saw $0.0$ then $1.0$ . Put together, $h_2$ depends on all three inputs even though each half saw only two. The diagram draws these same six numbers and the three joined pairs.

Code

The point is the check: run the same scalar rule both ways and confirm the hand arithmetic.

1import math2
3def forward_pass(xs: list[float], w_x: float, w_h: float) -> list[float]:4    h: float = 0.05    out: list[float] = []6    for x in xs:7        h = math.tanh(w_x * x + w_h * h)8        out.append(round(h, 2))9    return out10
11xs: list[float] = [1.0, 0.0, 1.0]12fwd: list[float] = forward_pass(xs, 0.8, 0.5)13bwd: list[float] = list(reversed(forward_pass(list(reversed(xs)), 0.8, 0.5)))14concat: list[tuple[float, float]] = [(f, b) for f, b in zip(fwd, bwd)]15
16print(fwd)  # -> [0.66, 0.32, 0.74]17print(bwd)  # -> [0.74, 0.32, 0.66]18print(concat[1])  # -> (0.32, 0.32)

The printout matches the hand computation: forward $[0.66, 0.32, 0.74]$ , backward $[0.74, 0.32, 0.66]$ , middle pair $(0.32, 0.32)$ .

Watch Out For

Running a backward pass while the input is still arriving

Symptom: a tagger scores well on full sentences in training, then stumbles live because each new token rewrites earlier backward states and forces a full recompute. Fix: keep bidirectional layers in encoders and labelers over complete input such as named entity recognition, and keep any generator or live decoder one-directional as in Sequence-to-Sequence decoders.

Feeding a doubled vector into a single-width layer

Symptom: the first batch fails with a shape mismatch because a linear layer built for 64 numbers receives 128 after concatenation. Fix: size every downstream input for the doubled width, and budget roughly twice the recurrent parameters with twice the per-step compute before you switch.

The Quick Version

A bidirectional RNN runs two independent recurrent passes with separate weights, one left to right and one right to left.
Each position hands downstream one concatenated pair, so the worked example gives $[0.66 ; 0.74]$ , $[0.32 ; 0.32]$ and $[0.74 ; 0.66]$ .
The middle pair sees the full three-value input because its forward $0.32$ covers the left two values and its backward $0.32$ covers the right two values.
A backward pass needs the last token before it can start, so live generation and streaming input cannot use a bidirectional layer.
Recurrent parameters, per-step compute and output width roughly double, so the next layer must accept the doubled width.