Neural Machine Translation (Attention)
Introduced the Attention mechanism, allowing Sequence-to-Sequence models to dynamically 'look back' at relevant parts of the input sentence instead of relying on a single context vector.
Paper: Neural Machine Translation by Jointly Learning to Align and Translate
Authors: Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio · 2014
Read the paper
The Problem
The Sequence-to-Sequence (Seq2Seq) model was a breakthrough, but it had a severe bottleneck. The Encoder had to compress the entire input sentence into a single, fixed-length vector. While this worked for short sentences, the model's performance collapsed on long sentences because a single vector simply couldn't hold that much information.
The Idea
When a human translates a sentence, they don't memorize the whole paragraph, close their eyes, and write the translation. They look back and forth, focusing on the specific source words that are relevant to the word they are currently translating. The authors mathematically formalized this 'looking back' as an Attention mechanism.
How It Works
Instead of just passing the final hidden state of the Encoder to the Decoder, the model makes all the intermediate hidden states of the Encoder available to the Decoder.
At every step of generating an output word, the Decoder looks at its current state and compares it against all the Encoder states. It calculates an 'alignment score' for each source word, representing how relevant that source word is right now. These scores are turned into probabilities (weights) using a softmax function. The model then creates a dynamic context vector by taking a weighted sum of the Encoder states, focusing heavily on the relevant words and ignoring the rest.
Why It Mattered
Attention completely solved the long-sentence degradation problem in Seq2Seq models. It allowed networks to easily handle sequences of almost any length. Furthermore, it provided the first real form of 'interpretability' in deep NLP: you could look at the attention weights to see exactly which source words the model was looking at when generating a specific output word.
What Came After
Attention became the most important concept in modern deep learning. While this paper used it alongside RNNs, researchers quickly realized that attention was powerful enough to stand on its own. This realization directly led to the Attention Is All You Need paper in 2017, which threw away the RNNs entirely to create the Transformer.