Bias–Variance Tradeoff

Expected error splits into rigid bias, jumpy variance and noise you cannot remove, and adding capacity lowers the first while raising the second.

Total error adds bias squared plus variance plus noise, so the middle model wins at 0.18 against 0.90 and 0.70

Why Does This Exist?

You train on fourteen cold days and nail heating demand, then a mild week misses by a mile. One gap, three possible causes, and each asks for the opposite fix. Don't guess. Name the piece first, then treat it.

Expected error is the average over fresh samples. Draw fourteen days, fit, measure squared error at one fixed temperature, repeat many times, average. You'll lean on expectation, variance and covariance for those summaries, so read that first if the symbols feel new.

This page covers the tradeoff choice itself. Overfitting and underfitting shows how to read it off training and validation curves. The proof with its own numbers lives in bias-variance decomposition. When the fix might be more rows, learning curves settle that call.

Think of It Like This

Darts on a board that moves in the wind

You watch three friends throw at the same bullseye in a pub.

One groups tight, five centimeters left of center, every round. Same miss, same spot. That is bias. The throw is consistently wrong, and averaging ten throws changes nothing because the offset never moves.

The second sprays around the bullseye. Left one round, right the next, high the next. Average a hundred and you land near center, but game night gives you one throw. That is variance. No steady offset, yet no single throw you'd trust.

Then someone opens the window and the board sways a centimeter either way. No throw removes that wobble. That is noise, and it sets the floor every player shares.

Where it stops: real training sets don't come back for another round, so you estimate these piles from splits rather than measuring them straight.

How It Actually Works

Fix one input, say demand at 30 degrees. Call the truth $f(x)$ , one number. Your procedure predicts $\hat{f}(x)$ , which jumps around because the fourteen days were random. Labels carry noise, so $y = f(x) + \epsilon$ where $\epsilon$ has mean zero and variance $\sigma^2$ .

In plain words: $x$ is the input you fix, $f(x)$ is the true value there, $\hat{f}(x)$ is what one random sample produces, $\mathbb{E}$ averages over all such samples, $\operatorname{Var}$ measures how those predictions scatter, $\epsilon$ is label noise, and $\sigma^2$ is its size.

For squared loss the average splits exactly:

\mathbb{E}[(y - \hat{f}(x))^2] = (\mathbb{E}[\hat{f}(x)] - f(x))^2 + \operatorname{Var}[\hat{f}(x)] + \sigma^2

The first term is squared bias, how far the average prediction sits from truth. The second is variance, how much one sample jumps. The third is noise nobody fits. That identity is algebra, not advice. Add and subtract $\mathbb{E}[\hat{f}(x)]$ inside the square, expand, and the cross term averages to zero.

The three terms

Squared bias punishes the wrong shape. A straight line on a curve misses in the same direction on most samples, so its average stays off truth. You will see both errors high and close together.

Variance punishes instability. A flexible curve threads each sample closely, so its average can sit near truth while any single fit swings hard. You will see training error low and held out error high.

Noise never enters the tradeoff. If demand also depends on a factory schedule you have no column for, no shape recovers it. Your job is the gap above that floor, and which movable term makes it up.

Worked numbers from three models

Take noise variance $\sigma^2 = 0.05$ for all three. A rigid line misses by 0.9 on average, so squared bias is $0.9^2 = 0.81$ , and it barely moves across samples, variance $0.04$ :

0.81 + 0.04 + 0.05 = 0.90

A parabola that matches the curve misses by 0.2 on average, so $0.2^2 = 0.04$ , with variance $0.09$ :

0.04 + 0.09 + 0.05 = 0.18

A degree 8 curve threads every point, bias $0.1$ so $0.1^2 = 0.01$ , but it whips across samples, variance $0.64$ :

0.01 + 0.64 + 0.05 = 0.70

The diagram draws these same three totals: 0.90 on the left, 0.18 in the middle, 0.70 on the right. Bias falls 0.81 to 0.04 to 0.01 while variance climbs 0.04 to 0.09 to 0.64. Same noise throughout.

Why the sum forms a U

Bias falls as you add capacity. Variance rises. Their sum drops while bias dominates, bottoms out, then climbs while variance dominates. That U is the picture, and its lowest point is the complexity worth shipping.

Two consequences follow. The minimum is not at zero bias, so a model that fits training data perfectly is rarely the best one. And the minimum moves right as you get more data, because data cuts variance for free and frees capacity to spend on bias. That is the honest reason more rows often help.

Code

The point is the check: recompute the three totals and the scatter of three fits.

1import numpy as np2
3
4def total(bias2: float, var: float, noise: float) -> float:5    return bias2 + var + noise6
7
8models: list[tuple[str, float, float]] = [9    ("simple", 0.81, 0.04),10    ("middle", 0.04, 0.09),11    ("complex", 0.01, 0.64),12]13noise: float = 0.0514for name, b2, v in models:15    print(f"{name} {total(b2, v, noise):.2f}")16    # -> simple 0.9017    # -> middle 0.1818    # -> complex 0.7019
20fits: np.ndarray = np.array([7.0, 8.0, 9.0])21mean: float = float(fits.mean())22var: float = float(((fits - mean) ** 2).mean())23print(f"{mean:.1f} {var:.2f} {var + noise:.2f}")  # -> 8.0 0.67 0.72

The printout matches the hand sums above: 0.90, 0.18, 0.70, plus three fits averaging 8.0 with variance 0.67 and 0.72 once noise joins.

Watch Out For

Judging bias and variance from one split

Symptom: one training split looks clean and one validation number picks a winner by 0.3 points, so you ship it. Next week the ranking flips. Every term in the split is an average over datasets, and one split is one draw.

Fix: use repeated cross-validation and report the spread beside the mean. If two models sit inside one standard error of each other, you have a tie, not a winner. That spread regularly erases gaps people had already written up.

Treating the U as a law of nature

Symptom: you cap a large model early because the curve must turn up, yet bigger models keep improving on held out data. You left accuracy behind to obey a picture.

Fix: remember the identity holds, but parameter count stops proxying variance once models interpolate. Optimizer preference for smooth fits acts as restraint, so capacity and effective capacity come apart. Use the split to diagnose what your error is made of, then treat with ridge regression or more data instead of counting parameters.

The Quick Version

Expected squared error equals squared bias plus variance plus noise, with no cross terms left over.
Bias is a steady miss shared across samples. Variance is scatter across samples around their own average.
Our sweep totals 0.90 for the rigid model, 0.18 for the middle, 0.70 for the flexible one, so the middle wins by 0.72 and 0.52.
Noise at 0.05 is a floor. No shape earns error below it on these points.
More data pushes the sweet spot toward higher capacity because it cuts variance for free.
Diagnose before treating. Both errors high and close means add capacity. Training low with held out high means restrain capacity or add rows.