Paper breakdown

Sequence to Sequence Learning

Introduced the Seq2Seq encoder-decoder architecture, allowing neural networks to map input sequences to output sequences of entirely different lengths.

Paper: Sequence to Sequence Learning with Neural Networks

Authors: Ilya Sutskever, Oriol Vinyals, Quoc V. Le · 2014

Read the paper

The Seq2Seq architecture uses an Encoder RNN to compress a sentence into a context vector, and a Decoder RNN to generate the translated sentence.

The Problem

Deep Neural Networks were excellent at mapping a fixed-size input to a fixed-size output (e.g., an image to a single class label). However, many of the world's most important problems, like language translation, speech recognition, and dialogue, require mapping a sequence of varying length to another sequence of a different varying length. Standard neural networks simply couldn't handle this mismatch.

The Idea

The authors proposed a two-network architecture. The first network (the Encoder) reads the input sequence one word at a time and compresses the entire meaning of the sentence into a single, fixed-dimensional vector. The second network (the Decoder) takes that single vector and unpacks it into the output sequence, one word at a time.

How It Works

The model uses Long Short-Term Memory (LSTM) networks, a type of RNN.

Encoder: It processes the input sentence (e.g., 'Hello world') token by token. Its final hidden state serves as the 'context vector', theoretically containing the semantic meaning of the entire sentence. Decoder: It is initialized with this context vector. It then begins generating the output sentence (e.g., 'Bonjour le monde') one word at a time. At each step, it uses its previous output as the input for the next step, stopping when it generates an <EOS> (End of Sentence) token.

The authors also discovered a bizarre but highly effective trick: if they fed the input sentence into the Encoder backwards, the model's translation accuracy skyrocketed, because it placed the start of the input sentence closer to the start of the output sentence.

Why It Mattered

Seq2Seq was a paradigm shift. It proved that a single end-to-end neural network could achieve state-of-the-art results on complex translation tasks, replacing decades of messy, rule-based statistical translation systems. It established the Encoder-Decoder architecture that dominates generative AI.

What Came After

While revolutionary, the basic Seq2Seq model had a flaw: forcing an entire paragraph into a single, fixed-size context vector caused the model to 'forget' early parts of the sentence. This bottleneck directly led to the invention of Attention mechanisms later that same year.