Lottery Ticket Hypothesis
Inside a big random network hides a small subnetwork that alone trains to full accuracy, if you rewind it to its lucky start.
Why Does This Exist?
Train a ResNet, prune 85 percent, retrain from scratch at that sparsity, and accuracy drops. Yet the unstructured sparse page keeps accuracy by healing survivors. The lottery ticket hypothesis explains why: the dense start held a lucky subnetwork, a winning ticket, that was trainable in isolation all along.
Compression frames it. Model pruning finds small models. This page is the ticket routine: iterative pruning, rewind to early weights, retrain the mask alone, and test if it matches dense.
Basics first: a ticket is a mask plus an initialization. Random reinit of the same mask usually fails. The start values matter, not just the shape.
Think of It Like This
A winning scratch card in a roll
A roll of scratch cards holds one big winner mixed with losers. You cannot see it until you scratch. Training scratches the roll. Pruning reveals which card won. Rewinding returns that card unscratched and proves it wins again. The analogy stops at reuse: cards work once, while tickets retrain reliably from the same start seed.
How It Actually Works
Train dense to convergence. Prune lowest 20 percent by magnitude. Rewind surviving weights to their values at epoch 3, not random anew. Retrain only survivors. Repeat prune and rewind five rounds to reach 15 percent density. Compare against same mask with fresh random init to prove the start carried the win.
Late rewinding is the assumption. Rewinding to step zero fails on deep nets. Rewinding to a few percent of training, when signs stabilize, succeeds. Track ticket accuracy across rounds to find the sparsity cliff.
Worked ticket run
Dense ResNet 18 scores 75.8 top 1 on a 100 class task. After five rounds, 15 percent remains: 1.8 of 12 million weights since 12 times 0.15 equals 1.8. Ticket rewound to epoch 3 retrains to 75.5, a 0.3 gap. Same mask randomly reinit scores 71.2, a 4.3 gap that proves initialization won, not shape alone.
Watch Out For
Claiming tickets transfer without retesting
A ticket found on ImageNet style data can fail on medical texture where early features differ. The symptom is dense parity at home with 5 point drops abroad. Fix it by re finding tickets per domain and by reporting transfer gaps, not just source wins.
The Quick Version
- Winning tickets are sparse masks plus lucky early weights.
- Iterative prune plus rewind to epoch 3 beats random reinit.
- Our 15 percent ticket held 75.5 versus 75.8 dense.
- Random reinit of the same mask fell to 71.2.
- Refind tickets per domain or transfer claims mislead.