Core ML

Regularization

A set of gentle brakes that stop a model from memorizing its training data, so it learns the real pattern and still works on data it has never seen.

Through the same scatter of points, an unregularized model draws a wiggly curve that chases every point while a regularized model draws a smooth curve that captures the trend

Why Does This Exist?

A powerful model has a tempting shortcut: instead of learning the underlying pattern, it can just memorize the training examples. That gets a perfect score on data it has already seen and falls apart on anything new. This is overfitting, and it's the central failure mode in machine learning.

You can picture it as a curve threading through scattered data. The right model draws a smooth line that captures the trend. An overfit model draws a wild, wiggly curve that passes exactly through every training point, including the random noise. It looks flawless on the training set and useless in the real world.

Regularization is the family of techniques that push back. Each one adds a preference for simpler models, deliberately making it harder to memorize so the model is forced to generalize. It trades a little accuracy on the training set for a lot more reliability on data it hasn't seen, which is the only accuracy that matters.

Think of It Like This

The student who memorizes vs. the one who understands

Two students prep for an exam. One memorizes the exact answers to last year's practice test word for word. The other learns the concepts. On the practice test the memorizer looks like a genius, but hand them a new question and they're stuck. The one who understood does fine on questions they've never seen. Regularization is the teacher forcing the model to understand rather than memorize, by penalizing pure rote recall.

How It Actually Works

Most regularization works by adding a penalty for complexity to the loss function. The model now minimizes two things at once: prediction error and how complicated it is. Gradient descent then naturally prefers simpler solutions. The common tools:

L2 regularization (weight decay). Add the sum of squared weights to the loss. This nudges every weight toward small values, so no single feature can dominate. It's the most widely used form.
L1 regularization. Add the sum of absolute weights. This drives some weights all the way to zero, effectively selecting a subset of features and producing a sparse model.
Dropout. During training, randomly switch off a fraction of neurons each step. The network can't lean on any one neuron, so it builds redundant, robust representations.
Early stopping. Watch performance on a held-out validation set and stop training the moment it stops improving, before the model starts memorizing.

How a complexity penalty steers training

Step 1 of 3

Measure error

Compute the usual prediction loss on the training batch.

Show all steps

Measure error: Compute the usual prediction loss on the training batch.
Add a penalty: Add a term that grows with model complexity, like the sum of squared weights.
Minimize both: Gradient descent shrinks error and complexity together, favoring the simpler fit that generalizes.

Show Me the Code

1import numpy as np2
3
4def l2_penalized_loss(errors: np.ndarray, weights: np.ndarray, lam: float = 0.1) -> float:5    data_loss = float(np.mean(errors ** 2))  # how wrong the predictions are6    penalty = lam * float(np.sum(weights ** 2))  # how large the weights are7    return data_loss + penalty  # minimize both at once8
9
10errors = np.array([0.2, -0.1, 0.3])11small = np.array([0.4, 0.3])12large = np.array([4.0, 3.0])13print(round(l2_penalized_loss(errors, small), 3))  # -> lower, preferred14print(round(l2_penalized_loss(errors, large), 3))  # -> higher, penalized

The lam knob sets how hard the brakes press: zero means no regularization, large values force a very simple model.

Watch Out For

Too much regularization causes underfitting

Regularization fights overfitting, but overdo it and you swing to the opposite problem. Crank the penalty too high and the model becomes so simple it can't even capture the real pattern, doing poorly on both training and new data. The goal is balance, not maximum simplicity.

Tune the strength on validation data, never the test set

The regularization strength is a hyperparameter you choose, so you need a separate validation set to tune it. Tune it against your final test set and you've quietly leaked information, and your reported accuracy will be optimistic.

The Quick Version

Regularization prevents overfitting by favoring simpler models that generalize.
L2 (weight decay) shrinks weights; L1 zeroes some out for sparsity.
Dropout and early stopping regularize without touching the weights directly.
Most methods add a complexity penalty to the loss that gradient descent then minimizes.
Too little lets the model memorize; too much makes it underfit, so tune the strength on validation data.