BERT

The Problem

Before BERT, pre-trained language models like OpenAI's GPT were strictly unidirectional. They processed text left-to-right to predict the next word. While this autoregressive approach is excellent for generation, it is suboptimal for understanding tasks (like question answering or sentiment analysis) where the context after a word is just as important as the context before it.

Standard bidirectional approaches at the time (like ELMo) merely trained a forward RNN and a backward RNN separately and concatenated their representations. This was a shallow form of bidirectionality that failed to let the model look at both left and right context simultaneously at every layer.

The Idea

The Google researchers realized that to achieve deep bidirectionality, they could not use the standard "predict the next word" training objective, because a truly bidirectional model would be able to trivially "see the future" and predict the target word.

Their breakthrough was reviving the Cloze task as a pre-training objective: the Masked Language Model (MLM). They randomly masked out 15% of the words in a sequence and tasked the model with predicting only those masked words based on the surrounding context (both left and right). They also added a Next Sentence Prediction (NSP) task to help the model understand sentence-level relationships.

How It Works

BERT (Bidirectional Encoder Representations from Transformers) uses only the Encoder half of the original Transformer architecture.

During pre-training on massive text corpora (BooksCorpus and English Wikipedia), the model learns deep, contextualized representations of words. Because it is an encoder-only architecture, every token attends to every other token in the sequence simultaneously, achieving true bidirectionality.

To adapt BERT to a specific downstream task, researchers simply add a single, small task-specific layer on top of the pre-trained BERT model and fine-tune all parameters end-to-end. This unified architecture drastically simplified the process of achieving state-of-the-art results on diverse tasks.

Why It Mattered

BERT represented a paradigm shift in NLP. It proved that unsupervised pre-training followed by supervised fine-tuning could dominate almost every benchmark, from GLUE to SQuAD. It effectively rendered task-specific architectures obsolete; instead of designing a complex custom network for question answering, you could simply fine-tune BERT.

What Came After

BERT triggered an explosion of encoder-only models. RoBERTa (2019) improved upon BERT by removing the Next Sentence Prediction task and training longer on more data. ALBERT (2019) introduced parameter-reduction techniques. DeBERTa (2020) added disentangled attention.

However, in the era of massive generative models (LLMs), pure encoder architectures have largely been superseded by decoder-only models (like the GPT family) or encoder-decoder models (like T5) that reframe all tasks as text generation, rather than text classification.