Sequence-to-Sequence

An encoder squeezes an input sequence into one fixed-size vector, and a decoder generates an output sequence from that vector alone. Lengths in and out never have to match.

The encoder squeezes three input tokens into one context number, and the decoder generates two output steps from that number alone

Why Does This Exist?

Translation refuses the shape most networks expect. "How are you?" is three words in English and four in French, with no position-by-position mapping between them. A classifier that outputs one label per input token has nothing sensible to do here. Both sides are sequences, and they do not even share a length.

The 2014 fix splits the job across two recurrent networks instead of one. An encoder reads the entire input and compresses it into a single fixed-size vector. A separate decoder then generates the whole output, one token at a time, starting from that vector alone. Input length and output length decouple completely: the vector in the middle does not care how long either side was.

The one idea this page stands on is recurrence: a network that reads a sequence step by step, carrying a running memory. This page covers what happens when that memory becomes the only bridge between two sequences. The bridge's famous flaw led straight to attention, and the bidirectional RNN is the encoder most systems actually used.

Think of It Like This

Reading a report once, then summarising it with the pages closed

Read an incident report exactly once, close it, and write a summary from memory. No glancing back while you write. Whatever made it from the pages into your head is now the only material available. Anything you failed to retain simply is not there for the summary to draw on.

That is the encoder-decoder shape exactly. The encoder reads the whole input and produces one compressed memory. The decoder writes the entire output using nothing else. If the memory dropped something important, the decoder cannot go back and check, which is precisely the constraint the diagram is showing.

The analogy stops at the size of your head. Human memory degrades gracefully and reconstructs gist. A context vector degrades nothing gracefully: past a certain input length, early details are not fuzzier, they are gone, overwritten by later steps. The bottleneck is harsher than forgetting. It is erasure by arithmetic.

How It Actually Works

The encoder: compress, never preserve

The encoder is a recurrent network, in practice often a bidirectional RNN, run over the input $x_1, \ldots, x_n$ . Its final hidden state, $c = h_n$ , is the context vector: one fixed-size summary of everything read, whether $n$ was 5 tokens or 500. A 500 token input and a 5 token input land in a vector of the exact same size. Nothing reserves extra room for longer sequences.

The decoder: generate from that vector alone

The decoder is a second, separately weighted recurrent network. It starts from the context vector, commonly by setting its first hidden state to $c$ directly, then generates one token at a time:

s_t = \mathrm{RNN}(y_{t-1}, s_{t-1}), \qquad p(y_t \mid y_{<t}, x) = \mathrm{softmax}(W s_t)

In plain words: $s_t$ is the decoder state at step $t$ , $y_{t-1}$ is the previous output token fed back in, and $W$ maps the state to vocabulary scores. Generation continues until an end-of-sequence token appears. Notice what never appears: the input tokens $x_1, \ldots, x_n$ play no part in any decoding step. Everything the decoder knows arrived once, at the start, packed into $c$ .

Training: teacher forcing

During training the decoder is not fed its own shaky early guesses. It receives the true previous token from the target at every step instead, a trick called teacher forcing. Training stays stable and fast, but it opens a real train and test gap: at inference time there is no ground truth to feed back, so the decoder consumes its own predictions, and one early mistake can drag the rest of the sequence with it.

Worked example

Shrink the encoder to width 1 so every number fits on the page. Start the hidden state at $h_0 = 0$ and update with $h_t = \tanh(x_t + h_{t-1})$ . Feed the input $[1, 0, 1]$ . Step one: $h_1 = \tanh(1) \approx 0.76$ . Step two: $h_2 = \tanh(0 + 0.76) \approx 0.64$ . Step three: $h_3 = \tanh(1 + 0.64) \approx 0.93$ .

The context vector is $c = 0.93$ , a single number summarising three inputs. The decoder now runs its steps from 0.93 alone: it never sees the $[1, 0, 1]$ , it never learns there were three tokens, and stretching the input to 300 tokens would not change its signature at all. That blindness is the bottleneck, drawn to scale in miniature.

Watch Out For

Expecting long inputs to compress without loss

A fixed-size vector holds a fixed amount of information, and content genuinely vanishes as inputs grow. The original sequence-to-sequence papers watched translation quality fall as source sentences grew past a few dozen tokens, and more data does not fix it because the ceiling is architectural, not statistical.

That wall is what attention exists to remove: not by widening the vector, but by letting the decoder look back at every encoder state instead of one compressed summary. If your inputs run long, budget for attention from the start.

Validating with teacher forcing still switched on

A model trained purely with teacher forcing has never once recovered from its own mistake, since every training input was the correct previous token. At inference that guarantee disappears, and a single wrong early token can push generation somewhere the model never practised recovering from. The offline metric then reads better than the deployed behaviour, sometimes much better.

Always validate with the decoding procedure production will use, feeding predictions back in. If the score drops off a cliff between the two setups, you found exposure bias, not a modelling failure.

The Quick Version

Sequence-to-sequence pairs an encoder with a decoder, decoupling input length from output length entirely.
The encoder compresses the whole input into one fixed-size context vector. The decoder generates the whole output from that vector alone.
No decoding step re-reads the input. Only the context vector and the decoder's own prior outputs feed each step.
Teacher forcing trains on true previous tokens, which opens a train and test gap worth measuring directly.
The fixed-size context is a genuine bottleneck, and quality degrades as inputs grow. Attention removes it by restoring lookback.