Lottery Ticket Hypothesis

The Problem

It was well known that after you train a massive neural network, you can "prune" it (delete 90% of the weights) and the network will still perform perfectly. This raised a huge paradox: if a model only needs 10% of its weights to function, why can't we just train a tiny 10% model from scratch and save massive amounts of compute? Every time researchers tried to train that tiny architecture from scratch, it failed to learn. It seemed you needed the massive, computationally expensive network just to facilitate the learning process.

The Idea

Frankle and Carbin proposed the Lottery Ticket Hypothesis. A massive neural network is essentially a giant bucket of random initializations (buying millions of lottery tickets). Within that giant network, there exists a tiny subnetwork (the "winning ticket") whose specific random initial weights were perfectly situated to learn the task. The rest of the network is mostly dead weight. You can train a tiny model from scratch, but only if it has the exact specific architecture and the exact specific starting weights of the winning ticket.

How It Works

They proved this using Iterative Magnitude Pruning (IMP):

1. Randomly initialize a large neural network (save these exact starting weights).
2. Train the network to completion.
3. Prune the network (delete the 80% of weights that are closest to zero). You now have a sparse subnetwork.
4. The Magic Step: Take that sparse subnetwork, and reset its remaining weights back to their original random values from Step 1.
5. Train this tiny, sparse subnetwork from scratch. It will learn faster and achieve higher accuracy than the original massive network.

If you reset the subnetwork with new random weights, it fails. The "winning ticket" is the combination of the sparse structure and its specific original initialization.

Why It Mattered

This paper fundamentally changed how we understand deep learning. It suggested that training massive networks is incredibly inefficient; we are wasting 90% of our compute just searching for the lucky subnetwork. It launched a massive sub-field dedicated to finding the "winning tickets" before training, aiming to drastically reduce the cost of AI.

What Came After

While finding winning tickets at initialization remains an unsolved Holy Grail for massive models, the hypothesis drove massive advances in sparse training techniques and hardware optimization for sparsity.