Hyperparameter Tuning
Training sets the weights, but you set the knobs around training such as depth and learning rate. Tuning is the search over those knob settings scored on data the model did not train on.
Why Does This Exist?
Every model has knobs training does not set: tree depth, learning rate, penalty strength. Miss one by a factor of ten and a good algorithm loses to a mediocre one tuned properly.
Carry one case down the page. You tune a churn model with three knobs: learning rate, maximum depth, and leaf count. Good values depend on each other in ways no formula gives you. A validation split scores any one setting honestly, as cross-validation schemes lay out. What it never tells you is which setting to try next, and three knobs at five values each already mean 125 combinations.
That search over settings, scored off the training data, is what hyperparameter tuning names. The sequential strategy that models the search itself lives in Bayesian optimization. What happens when the tuning leaks into the final score lives in overfitting and underfitting.
Think of It Like This
Seasoning soup one spoonful at a time
You season a large pot with salt, acid, and heat, and each attempt costs one tasting spoon. Salt matters enormously. The exact brand of black pepper barely moves the taste, but you do not know that going in.
Grid search decides every spoonful on paper before tasting: five salt levels crossed with five pepper levels on an even lattice, whether or not the first tastings already showed pepper was a waste. Random search throws both amounts independently each time, so across 25 spoonfuls salt still gets tried at 25 levels even though pepper ate a fifth of the grid attempts for nothing. The cook who remembers every taste and picks the next spoonful to resolve the most useful uncertainty is the Bayesian strategy.
Where it stops: soup lets you taste and adjust the same pot. Models do not adjust. Each spoonful is a full retrain from scratch, so a wasted evaluation costs hours, not a sip.
How It Actually Works
A hyperparameter is any setting fixed before training starts, meaning learning rate, depth, penalty, and neighbour count. A parameter is anything training updates, meaning weights and biases. Tuning searches the first group by training many models and scoring each on validation data.
Worked example: one knob matters, one is a decoy
Score depends almost entirely on log learning rate, with optimum at . Depth barely moves it. A 5 by 5 grid uses learning rates . The closest grid value to is away:
Twenty-five random draws from the same range land one draw from the optimum, more than ten times closer. The grid spent 25 runs to test 5 distinct learning rates. Random search spent the same 25 runs to test 25 distinct learning rates, because every draw is independent. When only some knobs matter, and you never know which in advance, the independent draws waste no resolution on the decoy axis.
Bayesian optimization remembers every score
Grid and random search pick every point blind to earlier results. Bayesian optimization fits a surrogate model, often a Gaussian process, to the knob-and-score pairs measured so far, then picks the next knob setting by balancing a high predicted score against high uncertainty. Each real evaluation costs a full training run, so spending a little compute on the surrogate pays off once training is slow.
Successive halving kills losers cheaply
A different lever: spend less on candidates that are clearly losing. Start 8 settings on a small budget, keep the best 4, double their budget, and repeat. Eight full trainings would cost 64 units. Halving in rounds of costs 32 units for the same winner in the common case, because a setting that is bad after one epoch rarely becomes the best after one hundred.
Code
The point is resolution per run: same budget, same range, independent draws cover the important axis more densely.
import random
rng = random.Random(9)grid: list[float] = [-4.0, -3.0, -2.0, -1.0, 0.0]draws: list[float] = [rng.uniform(-4.0, 0.0) for _ in range(25)]best: float = -1.5
grid_gap: float = min(abs(v - best) for v in grid)rand_gap: float = min(abs(v - best) for v in draws)print(round(grid_gap, 2), round(rand_gap, 2)) # -> 0.5 0.03print(len(set(grid)), len(draws)) # -> 5 25The printout matches the hand computation: the grid is from the optimum across 5 distinct values, random search is away across 25 distinct values. Change the seed from 9 to another integer and random search still tests 25 distinct values while the grid still tests 5.
Watch Out For
Tuning against the test set
The symptom appears months later: development scores looked good and production quietly misses by more than the reported error bars. It happens when the test set gets checked repeatedly while choosing knobs. Each check is a small peek, and forty peeks turn the test set into a second training set.
Symptom: validation and test agree during tuning, then fresh data disagrees. Fix: tune only on validation splits or cross-validation schemes, and touch the test set once after every knob decision is final.
Shipping a winner found at the edge of the range
The search returns its best learning rate sitting exactly at the largest value allowed. That result reports that the range was too narrow, not that it found an answer.
Symptom: the winning value equals a boundary you set, and neighbours inside the range score worse in a clear trend toward the wall. Fix: widen the range past the winner and rerun rather than shipping a boundary as an optimum. Read the failure as tuning debt, the same way overfitting and underfitting reads a gap between splits.
The Quick Version
- Hyperparameters are set before training, scored on validation data. Parameters are set by training.
- With 25 runs a 5 by 5 grid tests 5 learning rates while random search tests 25, landing 0.03 from the optimum against 0.50.
- Bayesian optimization fits a surrogate to past scores and picks the next setting by trading predicted score against uncertainty.
- Successive halving starts many settings cheaply, keeps winners, and spends the full budget only on survivors.
- Tune on validation only. A winner sitting at the range edge means the range was wrong, not that the search finished.