Chinchilla Scaling Laws
The 2022 DeepMind paper that redefined how LLMs are scaled, proving that most existing models were too large and under-trained on too little data.
Paper: Training Compute-Optimal Large Language Models
Authors: Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Arthur Brock, Oriol Vinyals, Jack W. Rae, Laurent Sifre · 2022
Read the paper
The Problem
In 2020, OpenAI published the original "Scaling Laws," which suggested that the best way to use extra compute was to drastically increase the size of the neural network (number of parameters), while only slightly increasing the amount of training data. This led to an arms race of massive models, peaking at the 530-billion parameter Megatron-Turing NLG. These models were incredibly expensive to run (inference) because of their sheer size, but researchers assumed this was the optimal way to train.
The Idea
DeepMind researchers rigorously re-evaluated scaling laws by training 400 different models with varying parameter counts and dataset sizes. They discovered that the original OpenAI scaling laws were wrong. For every 10x increase in compute budget, you should not just make the model 10x bigger; you should make the model ~3.16x bigger and train it on ~3.16x more data. They formulated the "Chinchilla Law": parameter count and training tokens should scale equally (a 1:1 ratio). Specifically, for a compute-optimal model, you should use roughly 20 training tokens for every 1 parameter.
How It Works
To prove their theory, DeepMind trained a new model called Chinchilla.
They compared it to their previous flagship model, Gopher.
- Gopher: 280 Billion parameters, trained on 300 Billion tokens.
- Chinchilla: 70 Billion parameters, trained on 1.4 Trillion tokens.
Both models used the exact same amount of training compute (FLOPs). However, Chinchilla (despite being 4x smaller) comprehensively outperformed Gopher on nearly every benchmark. Furthermore, because Chinchilla was smaller, it was vastly cheaper and faster to run in production.
Why It Mattered
The Chinchilla paper completely reset the AI industry's trajectory. It halted the race for trillion-parameter models and shifted the focus entirely to data acquisition. It proved that models like GPT-3 (175B parameters, 300B tokens) were severely under-trained.
What Came After
Almost every major LLM released after 2022 adhered to or exceeded Chinchilla scaling laws. LLaMA (65B parameters, 1.4T tokens) was a direct application of this theory. Today, companies "over-train" models significantly past the Chinchilla optimal point (e.g., Llama 3 8B trained on 15T tokens) because inference costs matter more than training compute.