Weight Decay
A crucial regularization technique built directly into optimization algorithms that penalizes large parameter values by shrinking them slightly every step.
Think of It Like This
Like a gardener continuously pruning the wildest, fastest-growing branches of a bush to ensure it maintains a balanced and healthy shape.
In modern optimizers like AdamW, weight decay directly subtracts a tiny fraction of the weight itself before applying the gradient update. This explicitly prevents any single neuron from growing too dominant and memorizing the training data. It is fundamentally equivalent to L2 regularization but applied algorithmically during the update step.