Switch Transformer
Scaled a sparse Mixture of Experts (MoE) model to a trillion parameters, proving that massive parameter scaling is possible without a proportional increase in compute.
Paper: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Authors: William Fedus, Barret Zoph, Noam Shazeer · 2021
Read the paper
The Problem
By 2021, the scaling laws were clear: bigger models perform better. However, making a model bigger (dense scaling) drastically increases both the memory footprint and the computational cost (FLOPs) required for every single token. Training a dense trillion-parameter model would require an absurd amount of compute, and running it in production would be impractically slow and expensive.
The Idea
The authors turned to a concept called Mixture of Experts (MoE), a form of sparse conditional computation.
Instead of passing every token through one massive Feed-Forward Network (FFN) layer, an MoE layer contains many smaller, specialized FFNs ("experts"). A trainable routing mechanism decides which expert(s) each token should be sent to.
While previous MoE models routed tokens to multiple experts (e.g., top-2 routing), the key innovation in the Switch Transformer was extreme simplification: route every token to exactly one expert.
How It Works
The architecture is based on T5. In the Transformer block, the standard dense FFN layer is replaced with a Switch Routing layer containing experts.
- Routing: A router network takes the token's representation and computes a probability distribution over the experts.
- Switching: The token is dispatched only to the single expert with the highest probability.
- Load Balancing: Left unchecked, a router might just send all tokens to one expert, rendering the others useless. The authors added a load-balancing loss penalty to force the router to distribute tokens evenly across all experts.
Because each token only activates one expert, the total parameter count can be scaled to over a trillion, but the active parameters (and FLOPs) per token remain roughly equivalent to a standard, much smaller dense model.
Why It Mattered
Switch Transformer successfully trained a 1.6-trillion-parameter model with comparable computational efficiency to a much smaller dense model. It proved that extreme sparsity was a viable path to circumventing the dense scaling wall. It also showed that pre-training speedups from sparsity carried over to fine-tuning on downstream tasks.
What Came After
While the 1.6T Switch Transformer was a landmark proof-of-concept, top-1 routing can sometimes lead to instability and suboptimal performance compared to activating slightly more experts.
The core sparse MoE concept introduced here has become central to the industry's most powerful models. The specific top-1 routing of Switch Transformer has largely been superseded by top-2 routing (used in Mixtral 8x7B) or more complex granular routing strategies (used in DeepSeek-V2 and V3), which strike a better balance between sparsity and representation power. It is also widely accepted (though officially unconfirmed) that OpenAI's GPT-4 is a massive sparse MoE model.