Skip to content
AI360XpertBeta

Gaussian Process Regression

Put a prior over whole functions instead of fixed weights, then let the data narrow it to curves that fit.

Mean curve passes through points at minus 2, 0, and 2 with std 0.001 at the points, 0.59 between them, and 0.99 far away at 4
Mean curve passes through points at minus 2, 0, and 2 with std 0.001 at the points, 0.59 between them, and 0.99 far away at 4

Why Does This Exist?

Fitting a curve usually means picking its shape first. Choose a line, a quadratic, a cubic, then fit weights. If the truth bends where your shape cannot, no amount of data fixes it. If you pick too flexible a shape, it chases noise.

A Gaussian process skips the shape choice. It places a prior directly over whole functions, infinitely many of them, scored by how smooth they are, then conditions that prior on the points you observed. The output at every input is a mean plus a variance, narrow where data pins it down and wide where nothing does.

That honesty is why small data science uses it. The surrogate inside bayesian optimization is usually this model, because optimisation needs to know where it is unsure. The point prediction habit it replaces lives in linear regression. The repair for overconfident numbers on familiar inputs lives in calibration.

Think of It Like This

Elastic strings pinned by pegs

Lay every smooth curve you can imagine as an elastic string across a board. The prior says some strings are more plausible than others, smoother ones more than wiggly ones.

Hammer pegs at three measured points. Throw away every string that misses a peg. What remains is a bundle that agrees tightly at the pegs and fans out in the gaps and far past the last peg.

That bundle is the predictive distribution. Its centre is the mean curve, its width is the uncertainty band, and the fan shape is not decoration. Tight at data and wide elsewhere is the whole claim.

Where it stops: real strings have fixed tension and pegs hold exactly. The model lets you set smoothness with a knob and lets pegs slip a little for noise. Change the knob and the same pegs give a different bundle.

How It Actually Works

Call the input xx and the unknown function value f(x)f(x). Call the kernel k(x,x′)k(x, x'), meaning how correlated two function values should be from inputs alone. The common RBF choice is:

k(x,x′)=exp⁡(−(x−x′)22ℓ2)k(x, x') = \exp\left(-\frac{(x - x')^2}{2\ell^2}\right)

ℓ\ell is the lengthscale, meaning how far you must move before the function can change a lot. Nearby inputs get correlation near one, distant inputs near zero. The kernel is the model. Picking it replaces picking a polynomial degree.

Treat values at any finite points as jointly normal with covariances from kk. Predicting is conditioning that joint normal on training outputs yy:

μ∗=K∗K−1y,Σ∗=K∗∗−K∗K−1K∗⊤\mu_* = K_* K^{-1} y, \qquad \Sigma_* = K_{**} - K_* K^{-1} K_*^\top

KK holds kernel values between training points, K∗K_* between query and training, K∗∗K_{**} between query points. μ∗\mu_* is the mean curve, the diagonal of Σ∗\Sigma_* is the variance. No gradient descent runs at prediction time. Uncertainty falls out of the same algebra as the mean.

Worked example: three pegs and one far query

Train at x=[−2.0,0.0,2.0]x = [-2.0, 0.0, 2.0] with y=[0.5,−1.0,0.8]y = [0.5, -1.0, 0.8] and ℓ=1.0\ell = 1.0. Kernel between points 2.0 apart is:

exp⁡(−2.0)≈0.1353\exp(-2.0) \approx 0.1353

Between points 4.0 apart it is exp⁡(−8.0)≈0.0003\exp(-8.0) \approx 0.0003. So the training matrix is nearly identity with 0.1353 off the centre diagonal and 0.0003 in the corners.

Condition on those three values with a tiny jitter 0.0000010.000001 for stability. At a training point the posterior standard deviation collapses to about 0.00100.0010 because the model trusts what it saw. Between pegs at x=1.0x = 1.0 it rises to about 0.590.59 because two smooth curves can disagree there and still hit both sides. Far away at x=4.0x = 4.0, correlation to every train point is near zero, so variance returns almost to the prior value of 1.01.0 and standard deviation reads about 0.990.99. The diagram draws exactly this: mean through three dots, pinched band at each dot, wider band between, widest band past the edge.

Why the bill is cubic

The formula contains K−1K^{-1}, the inverse of an nn by nn matrix. That costs order n3n^3 time. A few hundred points run instantly, a few thousand strain, tens of thousands are out of reach without sparse approximations. That wall is why this model rules small high value design and never serves web scale directly.

Code

The point is the check: rebuild the kernel numbers and the far field honesty with plain arithmetic.

import math
def rbf(a: float, b: float, ls: float = 1.0) -> float:    return math.exp(-((a - b) ** 2) / (2.0 * ls * ls))
k2: float = rbf(0.0, 2.0)k4: float = rbf(0.0, 4.0)print(f"{k2:.4f} {k4:.4f}")# -> 0.1353 0.0003
# single training point at 0.0 with value 1.0, jitter 1e-6y: float = 1.0k_star_far: float = rbf(4.0, 0.0)mean_far: float = k_star_far * y / (1.0 + 1e-6)var_far: float = 1.0 - k_star_far * k_star_far / (1.0 + 1e-6)print(f"{mean_far:.4f} {math.sqrt(var_far):.4f}")# -> 0.0003 1.0000

Two apart gives 0.1353, four apart gives 0.0003, matching the hand values. Far from data the mean decays to 0.0003 and the standard deviation returns to 1.0000, which is the prior admitting it learned nothing there.

Watch Out For

Running a full process on tens of thousands of rows

The O(n3)O(n^3) inverse sets in well before big data. A few thousand points already feel slow, and tens of thousands stall without inducing point or sparse tricks.

Symptom: training that never finishes while a simpler model flies, on data where uncertainty bands were never the need. Fix: check size first, and reach for sparse approximations or a different model when rows are plentiful and every prediction is familiar.

Trusting bands from a default kernel

The RBF assumes smooth infinite wiggle free curves with one lengthscale everywhere. Real functions jump, oscillate, or change roughness across the range, and a wrong ℓ\ell makes confident narrow bands in the wrong place.

Symptom: test points miss the 95 percent band far more than five percent of the time, often in one region. Fix: fit ℓ\ell and noise by marginal likelihood, plot the mean against held out points, and switch kernels when structure is periodic or rough instead of leaving the default.

The Quick Version

  • A Gaussian process is a prior over functions set entirely by a kernel k(x,x′)k(x, x').
  • RBF with lengthscale ℓ\ell says correlation falls as exp⁡(−d2/2ℓ2)\exp(-d^2 / 2\ell^2), so 2.0 apart gives 0.1353 and 4.0 apart gives 0.0003.
  • Prediction conditions a joint normal on training values, returning a mean and a variance together.
  • Our three pegs give std about 0.0010 at data, about 0.59 between, and about 0.99 far at 4.0.
  • Matrix inversion costs order n3n^3, which confines exact use to small and medium data.
  • Kernel and lengthscale are modelling choices, and wrong ones give confident wrong bands.