Gaussian Process Regression
Put a prior over whole functions instead of fixed weights, then let the data narrow it to curves that fit.
Why Does This Exist?
Fitting a curve usually means picking its shape first. Choose a line, a quadratic, a cubic, then fit weights. If the truth bends where your shape cannot, no amount of data fixes it. If you pick too flexible a shape, it chases noise.
A Gaussian process skips the shape choice. It places a prior directly over whole functions, infinitely many of them, scored by how smooth they are, then conditions that prior on the points you observed. The output at every input is a mean plus a variance, narrow where data pins it down and wide where nothing does.
That honesty is why small data science uses it. The surrogate inside bayesian optimization is usually this model, because optimisation needs to know where it is unsure. The point prediction habit it replaces lives in linear regression. The repair for overconfident numbers on familiar inputs lives in calibration.
Think of It Like This
Elastic strings pinned by pegs
Lay every smooth curve you can imagine as an elastic string across a board. The prior says some strings are more plausible than others, smoother ones more than wiggly ones.
Hammer pegs at three measured points. Throw away every string that misses a peg. What remains is a bundle that agrees tightly at the pegs and fans out in the gaps and far past the last peg.
That bundle is the predictive distribution. Its centre is the mean curve, its width is the uncertainty band, and the fan shape is not decoration. Tight at data and wide elsewhere is the whole claim.
Where it stops: real strings have fixed tension and pegs hold exactly. The model lets you set smoothness with a knob and lets pegs slip a little for noise. Change the knob and the same pegs give a different bundle.
How It Actually Works
Call the input and the unknown function value . Call the kernel , meaning how correlated two function values should be from inputs alone. The common RBF choice is:
is the lengthscale, meaning how far you must move before the function can change a lot. Nearby inputs get correlation near one, distant inputs near zero. The kernel is the model. Picking it replaces picking a polynomial degree.
Treat values at any finite points as jointly normal with covariances from . Predicting is conditioning that joint normal on training outputs :
holds kernel values between training points, between query and training, between query points. is the mean curve, the diagonal of is the variance. No gradient descent runs at prediction time. Uncertainty falls out of the same algebra as the mean.
Worked example: three pegs and one far query
Train at with and . Kernel between points 2.0 apart is:
Between points 4.0 apart it is . So the training matrix is nearly identity with 0.1353 off the centre diagonal and 0.0003 in the corners.
Condition on those three values with a tiny jitter for stability. At a training point the posterior standard deviation collapses to about because the model trusts what it saw. Between pegs at it rises to about because two smooth curves can disagree there and still hit both sides. Far away at , correlation to every train point is near zero, so variance returns almost to the prior value of and standard deviation reads about . The diagram draws exactly this: mean through three dots, pinched band at each dot, wider band between, widest band past the edge.
Why the bill is cubic
The formula contains , the inverse of an by matrix. That costs order time. A few hundred points run instantly, a few thousand strain, tens of thousands are out of reach without sparse approximations. That wall is why this model rules small high value design and never serves web scale directly.
Code
The point is the check: rebuild the kernel numbers and the far field honesty with plain arithmetic.
import math
def rbf(a: float, b: float, ls: float = 1.0) -> float: return math.exp(-((a - b) ** 2) / (2.0 * ls * ls))
k2: float = rbf(0.0, 2.0)k4: float = rbf(0.0, 4.0)print(f"{k2:.4f} {k4:.4f}")# -> 0.1353 0.0003
# single training point at 0.0 with value 1.0, jitter 1e-6y: float = 1.0k_star_far: float = rbf(4.0, 0.0)mean_far: float = k_star_far * y / (1.0 + 1e-6)var_far: float = 1.0 - k_star_far * k_star_far / (1.0 + 1e-6)print(f"{mean_far:.4f} {math.sqrt(var_far):.4f}")# -> 0.0003 1.0000Two apart gives 0.1353, four apart gives 0.0003, matching the hand values. Far from data the mean decays to 0.0003 and the standard deviation returns to 1.0000, which is the prior admitting it learned nothing there.
Watch Out For
Running a full process on tens of thousands of rows
The inverse sets in well before big data. A few thousand points already feel slow, and tens of thousands stall without inducing point or sparse tricks.
Symptom: training that never finishes while a simpler model flies, on data where uncertainty bands were never the need. Fix: check size first, and reach for sparse approximations or a different model when rows are plentiful and every prediction is familiar.
Trusting bands from a default kernel
The RBF assumes smooth infinite wiggle free curves with one lengthscale everywhere. Real functions jump, oscillate, or change roughness across the range, and a wrong makes confident narrow bands in the wrong place.
Symptom: test points miss the 95 percent band far more than five percent of the time, often in one region. Fix: fit and noise by marginal likelihood, plot the mean against held out points, and switch kernels when structure is periodic or rough instead of leaving the default.
The Quick Version
- A Gaussian process is a prior over functions set entirely by a kernel .
- RBF with lengthscale says correlation falls as , so 2.0 apart gives 0.1353 and 4.0 apart gives 0.0003.
- Prediction conditions a joint normal on training values, returning a mean and a variance together.
- Our three pegs give std about 0.0010 at data, about 0.59 between, and about 0.99 far at 4.0.
- Matrix inversion costs order , which confines exact use to small and medium data.
- Kernel and lengthscale are modelling choices, and wrong ones give confident wrong bands.