Skip to content
AI360XpertBeta

Bayesian Optimization

Each costly trial updates a cheap stand-in model that predicts both a score and its uncertainty everywhere, and the next trial goes where that map looks promising or genuinely unknown.

Three measured trials fit a surrogate mean with a band tight at each trial, and expected improvement picks candidate C with 0.45 over candidate A with 0.43
Three measured trials fit a surrogate mean with a band tight at each trial, and expected improvement picks candidate C with 0.45 over candidate A with 0.43

Why Does This Exist?

Some models take forty minutes to train once. When each hyperparameter setting costs a full run like that and your budget is maybe fifteen runs, the search method matters as much as the model. Hyperparameter tuning lays out the standard menu of grid search, random search, and successive halving, and all three choose points without learning from earlier results. Random search draws every learning rate independently, so run fourteen knows nothing about what runs one through thirteen just revealed.

That amnesia is the problem this page solves. After three runs you already know scores collapse below a certain rate and flatten above another, and anyone tuning by hand would aim the fourth run with that knowledge. Bayesian optimization is that aiming written as math: fit a cheap model to every (setting, score) pair measured so far, then let it nominate the next expensive trial.

This page covers the remember-and-nominate loop. The surrogate engine itself gets its full treatment at Gaussian processes, and how to read the costly runs once you have them is at learning curves. Two pieces of grounding before we start: a hyperparameter is a knob you set before training, like learning rate or tree depth, and the validation score is the number you are trying to push up, measured on data the model never trained on.

Think of It Like This

Prospecting for oil with one drill

You own one drill rig and a large field, and each borehole costs a fortune, so you've got about a dozen. After every hole you update a map of the whole field: ground near a strike probably holds more, ground near a dry hole probably holds less, and ground far from every hole could be anything.

Drilling only beside past strikes ignores vast unexplored ground that might be richer. Drilling only in the most unknown ground wastes fortunes confirming empty rock is empty. The prospector who wins drills where the map looks promising or genuinely unknown, and never where it looks mediocre and already understood.

The map is the surrogate model. The drill-here rule is the acquisition function.

Where it stops: real oil fields have geology with faults and traps, while the surrogate assumes nearby settings score similarly. When the true scores jump sharply between close settings, the map smooths over the jump and the drill rule trusts ground it should not.

How It Actually Works

Call the expensive thing ff, meaning the function from a hyperparameter setting xx to its validation score. Each real evaluation adds one pair to the history Dn={(x1,y1),…,(xn,yn)}D_n = \{(x_1, y_1), \dots, (x_n, y_n)\}, where yiy_i is the score measured for setting xix_i. Everything below reuses three names: μn(x)\mu_n(x) is the surrogate prediction at an untried setting xx, σn(x)\sigma_n(x) is its uncertainty there, and fbestf_{\text{best}} is the best score measured so far.

The surrogate: a cheap map with uncertainty bars

The surrogate is almost always a Gaussian process: a model fitted to DnD_n that is cheap to query at any xx, including settings nobody has tried. Unlike a plain regression curve it returns two numbers per point, the predicted score μn(x)\mu_n(x) and the uncertainty σn(x)\sigma_n(x). The uncertainty is tight near measured settings and wide in the gaps, exactly like the prospector map. Fitting it costs real compute, seconds to minutes as nn grows, which is why this whole method only pays when one real evaluation costs far more.

The acquisition step: turning belief into the next trial

The surrogate never says where to drill. The acquisition function turns μn(x)\mu_n(x) and σn(x)\sigma_n(x) into one ranking number per candidate, and the next real evaluation goes to the winner. The standard choice is expected improvement, the average gain over fbestf_{\text{best}} you'd get if the surrogate belief were true. Write Φ\Phi for the standard normal cumulative curve and ϕ\phi for its bell curve, with z=(μn(x)−fbest)/σn(x)z = (\mu_n(x) - f_{\text{best}}) / \sigma_n(x):

EI(x)=(μn(x)−fbest) Φ(z)+σn(x) ϕ(z)\mathrm{EI}(x) = (\mu_n(x) - f_{\text{best}}) \, \Phi(z) + \sigma_n(x) \, \phi(z)

The first term rewards a high predicted score and the second rewards high uncertainty, so a candidate wins by being promising, unknown, or a bit of both. Pure promising gets stuck beside the first lucky strike. Pure unknown never closes in.

Worked example: expected improvement picks C over A

Three trials are logged: settings 0.50.5, 2.02.0, and 3.53.5 scored 2.12.1, 5.05.0, and 3.23.2, so fbest=5.0f_{\text{best}} = 5.0. The surrogate offers two candidates. Candidate A sits near the best trial at x=2.2x = 2.2 with μ=5.4\mu = 5.4 and σ=0.4\sigma = 0.4. Candidate C sits in the wide gap at x=1.2x = 1.2 with μ=5.1\mu = 5.1 and σ=1.0\sigma = 1.0.

For A, the predicted gain is 5.4−5.0=0.45.4 - 5.0 = 0.4 and z=0.4/0.4=1.0z = 0.4 / 0.4 = 1.0. Normal tables give Φ(1.0)=0.8413\Phi(1.0) = 0.8413 and ϕ(1.0)=0.2420\phi(1.0) = 0.2420:

EI(A)=0.4×0.8413+0.4×0.2420=0.3365+0.0968=0.4333≈0.43\mathrm{EI}(A) = 0.4 \times 0.8413 + 0.4 \times 0.2420 = 0.3365 + 0.0968 = 0.4333 \approx 0.43

For C, the predicted gain is 5.1−5.0=0.15.1 - 5.0 = 0.1 and z=0.1/1.0=0.1z = 0.1 / 1.0 = 0.1. Tables give Φ(0.1)=0.5398\Phi(0.1) = 0.5398 and ϕ(0.1)=0.3970\phi(0.1) = 0.3970:

EI(C)=0.1×0.5398+1.0×0.3970=0.0540+0.3970=0.4510≈0.45\mathrm{EI}(C) = 0.1 \times 0.5398 + 1.0 \times 0.3970 = 0.0540 + 0.3970 = 0.4510 \approx 0.45

Since 0.45>0.430.45 > 0.43, the fourth trial goes to C. The diagram draws this exact verdict: three dots, a band pinched at each one, and the next-trial arrow on C. Notice what decided it: C predicts a smaller gain but its uncertainty term (0.39700.3970) dwarfs A (0.09680.0968), so the gap outvotes the near-certain small win.

Watch Out For

Running the machinery on a search that is cheap

Fitting the surrogate and maximizing expected improvement costs real wall-clock time every round. When one training run takes seconds, that overhead can exceed just trying a hundred random settings directly.

Symptom: the tuned search uses fewer trials than random search yet finishes later on the clock. Fix: check one evaluation cost against the surrogate fitting cost first, and below seconds per run use random search with successive halving instead.

Trusting the map built on two clustered points

Early in a search the surrogate has almost no signal, and a badly scaled kernel can make it confidently wrong far from the few points it has seen. The acquisition function then keeps nominating neighbours of one early lucky trial while the true optimum sits across an unexplored gap.

Symptom: after several rounds every suggested trial lands near the same spot and scores stop moving. Fix: seed the search with a spread of points across the whole range before letting expected improvement drive, and recheck the kernel scale if nominations cluster early.

The Quick Version

  • Bayesian optimization fits a cheap surrogate to every costly (setting, score) pair and asks it for a predicted score plus an uncertainty at untried settings.
  • Expected improvement ranks each candidate from predicted gain over the best score so far plus uncertainty, so here an uncertain 0.45 beats a near-certain 0.43.
  • The usual surrogate is a Gaussian process, whose uncertainty band pinches tight at measured trials and opens wide in the gaps.
  • The method repays its overhead only when one real evaluation costs far more than fitting the surrogate, like a forty-minute training run against seconds of math.
  • Seed the search with spread-out trials first, since a surrogate built on two or three clustered points can aim the next trial at the wrong place.