T5
Introduced T5, a framework that reframes every NLP task into a text-to-text format, enabling a single model architecture to handle diverse tasks.
Paper: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Authors: Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu · 2019
Read the paper
The Problem
By late 2019, the NLP landscape was a mess of different architectures and objectives. BERT used masked language modeling and an encoder-only architecture for classification. GPT-2 used autoregressive modeling and a decoder-only architecture for generation. Comparing techniques across these fundamentally different paradigms was extremely difficult, and deploying a single model to handle translation, summarization, and classification simultaneously was complex.
The Idea
The Google researchers proposed a radical unification: reframe every single NLP problem as a "text-to-text" task.
Instead of having different output heads for different tasks (e.g., a classification head for sentiment analysis, a generation head for translation), the model always takes text as input and generates text as output. For example, to classify sentiment, the input is `"sentiment: This movie is terrible"` and the target output is the literal string `"negative"`.
How It Works
To support this text-to-text framework, T5 (Text-to-Text Transfer Transformer) uses a standard encoder-decoder Transformer architecture.
The paper is essentially a massive empirical survey. The authors systematically tested dozens of architectural variations, pre-training objectives, datasets, and fine-tuning strategies to find the optimal combination.
They also introduced the C4 dataset (Colossal Clean Crawled Corpus), a massive cleaned version of Common Crawl, which became the standard pre-training dataset for the industry for years. Their winning pre-training objective was a span-corruption task: masking out continuous spans of text and forcing the decoder to reconstruct them.
Why It Mattered
T5 brought extreme rigor to the wild west of transfer learning. By standardizing the input-output format, it allowed for true apples-to-apples comparisons of different pre-training techniques. The text-to-text framework proved highly successful, and T5 models achieved state-of-the-art results across a massive suite of benchmarks.
It also popularized the practice of releasing a spectrum of model sizes (from T5-Small at 60M parameters to T5-11B), making powerful open-source models accessible to researchers with limited compute.
What Came After
T5's encoder-decoder architecture remained popular for tasks requiring heavy reasoning over input text (like translation and summarization) and spawned successors like FLAN-T5 (instruction-tuned T5). However, for general-purpose foundational models, the encoder-decoder architecture has largely been superseded by decoder-only models (the GPT/Llama paradigm) due to their superior efficiency in training and scaling.