Core ML

Sequence-to-Sequence

An encoder compresses an input sequence into one fixed-size vector, and a decoder generates an output sequence from that vector alone — the shape behind translation and summarization.

The encoder compresses an entire input sequence into one fixed-size vector, and the decoder must generate the whole output sequence from that single vector alone

Why Does This Exist?

Translation doesn't fit the "one input, one output" shape most networks are built around. "How are you?" is three words in English and four in French, and there's no position-by-position mapping between them — a classifier that outputs one label per input token has nothing sensible to do here. The input and output are both sequences, and they don't even share a length.

The 2014 fix reframes the whole problem around two recurrent networks instead of one. An encoder reads the entire input sequence and compresses it down into a single fixed-size vector. A separate decoder then generates the entire output sequence, one token at a time, using nothing but that one vector as its starting context. Input length and output length are now completely decoupled — the vector in the middle doesn't care how long either sequence was.

Think of It Like This

Reading a report, then writing a summary from memory alone

Imagine reading an entire incident report once, closing it, and then writing a summary from memory — no glancing back at the original while you write. Whatever made it from the report into your head is now the only thing available to you; anything you didn't retain simply isn't there for the summary to draw on.

That's the encoder-decoder shape exactly. The encoder reads the whole input and produces one compressed memory. The decoder writes the entire output using nothing else. If the memory left something important out, the decoder has no way to go back and check — which is exactly the constraint the diagram's caption is pointing at.

How It Actually Works

The encoder: compress, don't preserve

The encoder is a recurrent network — often a bidirectional LSTM or GRU in practice — run over the entire input sequence $x_1, \ldots, x_n$ . Its final hidden state, $c = h_n$ , is the context vector: one fixed-size summary of everything the encoder read, regardless of whether $n$ was 5 tokens or 500.

This is the step the diagram labels the bottleneck. A 500-token input and a 5-token input both get compressed into a vector of the exact same size. Nothing about the architecture reserves more room for a longer sequence — the encoder simply has to fit more into the same-sized container.

The decoder: generate, conditioned on that vector alone

The decoder is a second, separately-weighted recurrent network. It's initialized with the encoder's context vector — commonly by setting the decoder's first hidden state to $c$ directly — and then generates the output one token at a time:

s_t = \text{RNN}(y_{t-1}, s_{t-1}), \qquad p(y_t \mid y_{<t}, x) = \text{softmax}(W s_t)

Each generated token $y_{t-1}$ feeds back in as the input to produce the next one, and generation continues until the decoder produces an end-of-sequence token. Note what's not in that equation: the original input tokens $x_1, \ldots, x_n$ don't appear anywhere in the decoder's per-step computation. Everything the decoder knows about the input arrived once, at initialization, packed into $c$ .

Training: teacher forcing

During training, the decoder doesn't actually feed its own (possibly wrong, early in training) predictions back into itself — it's given the true previous token from the target sequence at every step instead, a technique called teacher forcing. This keeps training stable and parallelizable across positions, but it creates a real train/inference mismatch: at inference time there is no ground truth to feed back, so the decoder consumes its own predictions, and an early mistake can compound as the sequence continues.

Show Me the Code

A minimal encoder-decoder pass: one function compresses a sequence to a vector, a second generates from it.

1import numpy as np2
3
4def encode(sequence: list[np.ndarray], w: np.ndarray, hidden_size: int) -> np.ndarray:5    h = np.zeros(hidden_size)6    for x_t in sequence:                                 # read the whole input7        h = np.tanh(w @ np.concatenate([x_t, h]))8    return h                                              # the one vector the decoder gets9
10
11def decode(context: np.ndarray, w: np.ndarray, steps: int) -> list[np.ndarray]:12    outputs, s = [], context                              # decoder starts FROM the context13    y_prev = np.zeros_like(context)14    for _ in range(steps):                                # no access to the input sequence here15        s = np.tanh(w @ np.concatenate([y_prev, s]))16        outputs.append(s)17        y_prev = s18    return outputs19
20
21rng = np.random.default_rng(0)22w_enc, w_dec = rng.normal(scale=0.4, size=(4, 6)), rng.normal(scale=0.4, size=(4, 8))23context = encode([np.array([1.0, 0.0]), np.array([0.0, 1.0]), np.array([1.0, 1.0])], w_enc, 4)24decoded = decode(context, w_dec, steps=2)25print(len(decoded), decoded[0].shape)  # -> 2 (4,) — 2 output steps from a 3-step input, same context

decode never touches sequence — only context, the encoder's final hidden state. Change the input from 3 tokens to 300 and decode's signature doesn't change at all, because the bottleneck already threw away the length.

Watch Out For

Expecting long inputs to compress without loss

The context vector has a fixed size no matter the input length, and information genuinely gets lost as sequences grow — translation quality on this architecture drops measurably once source sentences pass roughly 20 to 30 tokens, and it's not a training issue that more data fixes. It's the architecture: a fixed-size vector has a fixed amount of room, and a longer sequence is being asked to fit through the exact same doorway.

This is the wall attention exists to remove — not by making the vector bigger, but by giving the decoder a way to look back at every encoder state instead of relying on one compressed summary.

Training with teacher forcing, evaluating without noticing the gap

A model trained purely with teacher forcing has never, during training, had to recover from one of its own mistakes — every input to the decoder was the correct previous token. At inference time that guarantee disappears, and a single wrong early token can push the rest of the generated sequence into a region the model never practiced recovering from, a failure mode sometimes called exposure bias. Validate with the same decoding procedure you'll actually use in production, not with teacher forcing turned on, or the offline metric will look better than the deployed behavior.

The Quick Version

Sequence-to-sequence pairs an encoder and a decoder, decoupling input length from output length entirely.
The encoder compresses the whole input into one fixed-size context vector; the decoder generates the whole output from that vector alone.
The decoder's per-step computation never re-reads the original input — only the context vector and its own prior outputs.
Teacher forcing trains the decoder on true previous tokens, which creates a train/inference mismatch worth validating against directly.
The fixed-size context vector is a genuine bottleneck: quality degrades measurably as input sequences get long.