BERT — bidirectional pretraining for language understanding
How masked-language-model pretraining gave encoders deep bidirectional context and reset the NLP benchmark board.
Paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Authors: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova · 2018
Read the paperThe problem it solved
Earlier language models read left-to-right, so a word's representation couldn't use the words that follow it. That one-directional context caps understanding tasks like question answering.
The key idea
Pretrain a transformer encoder with a masked language model objective: hide some tokens and train the model to predict them from both sides. This forces genuinely bidirectional representations.
What made it work
- Masked LM enables bidirectionality without letting a token "see itself".
- Next-sentence prediction adds a coarse sentence-relationship signal (later work questioned its value).
- Pretrain then fine-tune — one pretrained model adapts to many tasks with a small head.
Why it matters
BERT popularized the pretrain/fine-tune recipe for understanding tasks and topped a swath of benchmarks, making transfer learning the default in NLP.