Skip to content
AI360XpertBeta

Polynomial Regression

Add powers of the input as features and a straight-line solver can draw curves. Bend it too far and it memorizes noise instead of the trend.

Degree 1 draws a straight line with error 1.0 through curving points while degree 2 recovers the true parabola with zero error
Degree 1 draws a straight line with error 1.0 through curving points while degree 2 recovers the true parabola with zero error

Why Does This Exist?

A shipping team plots delivery delay against distance and the trend curves: delays climb faster than distance for a while, then level off. Linear regression draws one straight line through that bend and misses both ends no matter how the line tilts.

Two ideas make the rest click. Least squares finds the coefficients that minimize squared error on the given columns. Model capacity is how bendy the fitted shape is allowed to be. The straight-line machinery itself lives in linear regression.

This page covers the smallest fix: feed the solver powers of the input and let the same least squares fit a curve. The memorization trap that high degrees fall into lives in overfitting and underfitting. The formal tradeoff behind picking the degree lives in bias-variance tradeoff. The step-by-step visual lives in polynomial regression visualized.

Think of It Like This

Bending a wire to match dots on a table

A stiff straight wire laid on a table can only lie flat in one direction: no pushing makes it trace a curve through scattered dots. A wire with two deliberate bends can roughly follow a gentle arc through them.

A wire with a kink for nearly every dot can be forced to touch each dot exactly, yet between dots it zigzags wildly, tracing a shape the dots never suggested. More bends match what sits in front of you better, and past a point they stop tracking the real shape and start tracking where each dot happened to land.

Where it stops: you can see the wire zigzag and refuse it. A fitted polynomial reports only its error numbers, so without held-out checks the wildest wire looks like the best one.

How It Actually Works

Turning one feature into several

Given one input xx, degree dd builds the columns x,x2,…,xdx, x^2, \dots, x^d and fits ordinary least squares on top:

y^=w0+w1x+w2x2+⋯+wdxd\hat{y} = w_0 + w_1 x + w_2 x^2 + \dots + w_d x^d

The name says curve, the solver says line: w0,…,wdw_0, \dots, w_d come from the same normal equations as linear regression, applied to wider columns. Degree 1 is a straight line. Degree 2 adds one bend. Near as many coefficients as points, the curve can thread every point and training error hits zero.

Worked example: four points on a parabola

Truth is y=x2+1y = x^2 + 1 at x=0,1,2,3x = 0, 1, 2, 3, giving points (0,1),(1,2),(2,5),(3,10)(0,1), (1,2), (2,5), (3,10). Degree 2 least squares recovers weights exactly:

w=[1,0,1]MSE=0.0w = [1, 0, 1] \quad \mathrm{MSE} = 0.0

Degree 1 finds the line y=3xy = 3x, predicting 0,3,6,90, 3, 6, 9 against true 1,2,5,101, 2, 5, 10. Every error is ±1\pm 1, so:

MSE=(1+1+1+1)/4=1.0\mathrm{MSE} = (1 + 1 + 1 + 1) / 4 = 1.0

The bend is worth the full 1.0. That gap is the diagram headline: one straight line cutting across the arc against one parabola threading all four dots.

Why zero training error warns, not wins

A degree high enough to touch every point spends its flexibility on each point's noise. Between points the polynomial oscillates, wildly near the edges, and just past the training range powers explode: the curve that looks perfect inside is the one that generalizes worst outside. Never trust predictions past the fitted xx range.

Choosing degree honestly

Training error falls monotonically with degree, by construction, so it can never pick the degree. Sweep degrees against held-out validation error and take the smallest degree near the flat bottom. That is overfitting and underfitting with a single visible dial.

Code

The point is the check: fit degrees 1 and 2 on the worked example with plain least squares and confirm errors 1.0 and 0.0.

xs: list[float] = [0.0, 1.0, 2.0, 3.0]ys: list[float] = [1.0, 2.0, 5.0, 10.0]

def fit_degree(xs: list[float], ys: list[float], degree: int) -> list[float]:    """Least squares weights via normal equations and elimination."""    n: int = degree + 1    a: list[list[float]] = [[sum(x ** (j + k) for x in xs) for k in range(n)] for j in range(n)]    b: list[float] = [sum(y * (x**j) for x, y in zip(xs, ys)) for j in range(n)]    m: list[list[float]] = [a[i][:] + [b[i]] for i in range(n)]    for c in range(n):        p: int = max(range(c, n), key=lambda r: abs(m[r][c]))        m[c], m[p] = m[p], m[c]        for r in range(n):            if r != c:                f: float = m[r][c] / m[c][c]                for k in range(c, n + 1):                    m[r][k] -= f * m[c][k]    return [m[i][n] / m[i][i] for i in range(n)]

for degree in (1, 2):    w: list[float] = fit_degree(xs, ys, degree)    preds: list[float] = [sum(w[j] * (x**j) for j in range(degree + 1)) for x in xs]    mse: float = sum((p - y) ** 2 for p, y in zip(preds, ys)) / len(ys)    print(f"degree {degree}: {','.join(f'{v:.1f}' for v in w)} mse {mse:.1f}")    # -> degree 1: -0.0,3.0 mse 1.0    # -> degree 2: 1.0,-0.0,1.0 mse 0.0

Degree 2 recovers [1,0,1][1, 0, 1], the true x2+1x^2 + 1. Change one yy by +0.5+0.5 and refit: degree 2 barely moves while a degree 3 fit through four points would chase the noise exactly.

Watch Out For

Picking the degree by training error

Training error only ever falls as degree rises, since extra columns can always fit the training rows at least as well. Minimizing it picks the highest degree tried, regardless of generalization.

Symptom: the chosen degree threads every training dot while fresh data errors climb. Fix: select by held-out validation error and prefer the smallest degree near the flat bottom.

Trusting the curve past the training edge

Inside the fitted range a high-degree curve can look sane while just outside it the leading power dominates and predictions rocket. Polynomials have no reason to flatten past what they saw.

Symptom: sane interpolations beside absurd forecasts one step outside the data range. Fix: never extrapolate a polynomial fit; re-fit or switch methods when the range must grow.

The Quick Version

  • Polynomial regression fits ordinary least squares on powers of the input, so a linear solver draws curves.
  • Our four points on y=x2+1y = x^2 + 1 give degree 2 weights [1,0,1][1, 0, 1] with error 0.0 against degree 1 error 1.0.
  • Degree is a visible capacity dial: low underfits, high threads every point and oscillates between them.
  • Training error falls monotonically with degree and can never select it; held-out validation error does.
  • High-degree curves explode past the training range, so never extrapolate them.