Dropout
Introduced Dropout, a remarkably simple yet highly effective regularization technique that prevents neural networks from overfitting by randomly disabling neurons.
Paper: Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Authors: Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov · 2014
Read the paper
The Problem
Deep neural networks with millions of parameters are incredibly powerful, but they are highly prone to overfitting. They tend to memorize the training data, leading to poor generalization on unseen test data. The standard way to combat this was to train many different models and average their predictions (ensembling), but training and running dozens of massive deep neural networks was computationally prohibitive.
The Idea
The authors wanted to simulate the effect of training an ensemble of exponentially many different neural networks, but within a single model and a single training run. They realized that if they randomly disabled different parts of the network during each training step, the network could not rely on any single neuron or specific combination of neurons.
How It Works
During the forward pass of training, Dropout randomly 'drops out' (sets to zero) a proportion of neurons in a layer (typically 20% to 50%).
Because a neuron never knows which of its neighbors will be active during a given pass, it cannot rely on them to fix its mistakes. This breaks up 'co-adaptations' among neurons. Every neuron is forced to learn robust, useful features on its own that work well in conjunction with many different random subsets of other neurons.
During inference (testing), Dropout is turned off. All neurons are active, but their outgoing weights are scaled down by the dropout probability to account for the fact that more neurons are active than during training.
Why It Mattered
Dropout was elegantly simple to implement (just a few lines of code) and profoundly effective. It became the standard regularization technique for deep learning, allowing researchers to train much larger, more expressive models without crippling them with overfitting.
What Came After
Dropout remains widely used, especially in fully connected layers. However, modern architectures rely on it slightly less than they did in 2014. Techniques like Batch Normalization introduced their own regularization effects, and modern massive datasets (like those used for LLMs) naturally reduce overfitting simply by virtue of their scale.