Skip to content
AI360Xpert
Core ML
Visual explainer

Hyperparameter Tuning

Why random search beats grid search, how Bayesian optimisation learns from past runs, and why tuning on your test set invalidates your results.

Parameters are learned during training. Hyperparameters are chosen before training.
Parameters are learned during training. Hyperparameters are chosen before training.

Models have two kinds of settings. Parameters (like the weights in a neural network) are learned automatically from the data during training. Hyperparameters (like the learning rate, or the depth of a tree) govern how the learning happens. The model cannot learn them; you have to set them before you hit run.

The Grid Search Trap

Grid Search exhaustively checks every combination, wasting time on useless dimensions.
Grid Search exhaustively checks every combination, wasting time on useless dimensions.

The simplest way to find good hyperparameters is Grid Search: test every combination of values. But if one parameter matters (like learning rate) and another doesn't (like batch size), a 4x4 grid spends 16 runs but only tests 4 unique learning rates. It is horribly inefficient.

Why Random is Better

Random Search finds better values faster by not aligning on a rigid grid.
Random Search finds better values faster by not aligning on a rigid grid.

Random Search simply picks random values within a range. In 16 runs, it tests 16 completely unique values for the important parameter. It consistently finds better configurations in less time than Grid Search because it doesn't waste cycles stepping through a rigid alignment.

Bayesian Optimisation

Bayesian Optimisation learns from past runs to predict the next best guess.
Bayesian Optimisation learns from past runs to predict the next best guess.

Random search is blind. Bayesian Optimisation remembers. After a few runs, it builds a mathematical map (a surrogate model) predicting where the best parameters might be. It balances exploitation (testing near known good values) with exploration (testing areas of high uncertainty), drastically reducing the number of runs needed.

Where It Breaks

The failure: tuning on the validation set leaks into the final score.
The failure: tuning on the validation set leaks into the final score.

When you try 100 parameter combinations and pick the one that scores highest on your validation set, that score is no longer an honest estimate of real-world performance. You have overfit to the validation set. You must keep a completely separate test set locked away until tuning is entirely finished, or your reported accuracy is a lie.

The Quick Version

  • The goal: Find the best external settings (hyperparameters) for a model.
  • Grid Search: Tests every combination. Terribly slow and inefficient.
  • Random Search: Better than Grid Search because it explores more unique values per parameter.
  • Bayesian Optimisation: Uses past runs to intelligently guess the next best setting.
  • Failure: Tuning on the test set causes data leakage. You need Train, Validation (for tuning), and Test (for the final score).

What to Read Next