Polynomial Regression

When you fit a first-degree linear regression line $\hat{y} = w_1 x + b$ through non-linear data, the errors do not scatter randomly. In a diagnostic residual plot ($e_i = y_i - \hat{y}_i$ versus $x_i$), the residuals trace a distinct, systematic U-shaped arc:

$\mathbb{E}[e \mid x] \ne 0$

This non-random residual curvature is proof of structural model underfitting. The hypothesis space is too rigid to express the underlying curvature of the data-generating process. Gathering more training rows cannot resolve this failure; the feature space itself must be expanded.

Square Your Inputs

Polynomial regression bends the hypothesis curve without altering the linear solver. By applying a non-linear basis expansion $\phi(x)$, a scalar input $x$ is mapped into higher-order powers:

$\phi(x) = \begin{bmatrix} x^0 \\ x^1 \\ x^2 \end{bmatrix} = \begin{bmatrix} 1 \\ x \\ x^2 \end{bmatrix}$

For an input scalar $x = 2$, the expanded feature vector evaluates to $[1, 2, 4]$. The regression model computes:

$\hat{y} = w_0 (1) + w_1 (x) + w_2 (x^2) = \mathbf{w}^T \phi(x)$

While the relationship between $x$ and $\hat{y}$ is non-linear and curved, the model remains strictly linear in its parameters $\mathbf{w}$. Consequently, the exact closed-form Ordinary Least Squares (OLS) solver remains fully intact:

$\mathbf{w}^* = (\mathbf{\Phi}^T \mathbf{\Phi})^{-1} \mathbf{\Phi}^T \mathbf{y}$

Parabola Nails Three

Consider a data-generating function governed by $y = x^2$ observed at three sample points: $(1, 1)$, $(2, 4)$, and $(3, 9)$. Constructing the Vandermonde matrix $\mathbf{\Phi}$ and solving the normal equations yields exact parameter weights:

$\mathbf{w}^* = \begin{bmatrix} w_0 \\ w_1 \\ w_2 \end{bmatrix} = \begin{bmatrix} 0.0 \\ 0.0 \\ 1.0 \end{bmatrix} \implies \hat{y} = x^2$

Evaluating the fitted parabola at each knot:

$\hat{y}(1) = 1, \quad \hat{y}(2) = 4, \quad \hat{y}(3) = 9 \implies e_i = y_i - \hat{y}_i = 0.00$

The training mean squared error collapses to exactly $\text{MSE} = 0.000$. The linear solver has drawn a curved quadratic manifold through every point without requiring iterative numerical optimization.

Degrees Wobble Wildly

Increasing polynomial order expands model capacity, but unconstrained flexibility introduces severe instability. Across a ten-point dataset:

• Degree 1 (Stiff): Underfits, missing the true arc ($\text{MSE}_{\text{test}} = 0.45$).
• Degree 2 (Glides): Matches the true quadratic signal, smoothing out random noise ($\text{MSE}_{\text{test}} = 0.14$).
• Degree 9 (Waves): Threads every single training point with zero training error ($\text{MSE}_{\text{train}} = 0.00$), but oscillates violently between the knots.

This oscillatory pathology is known as Runge's phenomenon. Near the boundaries and sample intervals, high-degree polynomial derivatives swing wildly, turning interpolation into high-variance noise.

Sweet Degree Wins

Evaluating candidate polynomial degrees on an untouched held-out test set reveals the empirical cost of excess capacity:

$\text{Test MSE}_{d=1} = 0.45, \quad \text{Test MSE}_{d=2} = 0.14, \quad \text{Test MSE}_{d=9} = 0.62$

Degree 2 achieves a $69\%$ error reduction over the rigid linear model ($0.45 \to 0.14$). Conversely, degree 9 suffers a $+343\%$ error explosion ($0.14 \to 0.62$). Despite achieving a perfect training score of $0.00$, degree 9 collapses on fresh test data because its oscillatory excursions fail to generalize.

Pick Smallest Winner

Model complexity selection should follow the validation elbow criterion governed by Occam's razor:

1. The Validation Elbow: Trace validation loss across degree $d \in [1, \dots, 9]$. Validation loss drops steeply from $d=1$ ($0.45$) to $d=2$ ($0.14$), where marginal error gains flatten out ($d=3$ yields $0.145$).
2. Occam's Tiebreaker: When two candidate degrees yield nearly identical validation risk, always select the smaller degree ($d=2$).
3. Conditioning Safeguard: Higher powers of $x$ severely inflate the condition number $\kappa(\mathbf{\Phi}^T \mathbf{\Phi})$, causing collinearity and numeric instability in matrix inversion.

Where It Breaks

Polynomial models are strictly local interpolators. Outside the training domain $[x_{\min}, x_{\max}]$, the leading power term $w_d x^d$ dominates the function's asymptotic behavior:

$\lim_{|x| \to \infty} \hat{y}(x) \propto w_d x^d$

When evaluating a degree-9 model at $x = 6$ beyond a training horizon of $x \in [1, 4]$, the true physical trend continues a gentle linear drift ($y_{\text{true}} = 12$). However, the polynomial rockets vertically to $\hat{y} = 36$, creating a catastrophic $+24$ error shock. Never extrapolate high-degree polynomials beyond their observed domain boundaries; clamp inputs or transition to splines with linear boundary constraints.

The Quick Version

• Curved residual plots prove that linear models suffer from missing non-linear shape.
• Non-linear basis expansion maps $x = 2$ into $[1, 2, 4]$ while preserving OLS linearity in $\mathbf{w}$.
• A degree-2 parabola fits points $(1,1), (2,4), (3,9)$ with zero residual error.
• Unconstrained degree-9 polynomials suffer from Runge's phenomenon, oscillating wildly between dots.
• Degree 2 achieves the optimal test error ($0.14$) while degree 9 explodes ($0.62$).
• Choose the smallest degree at the validation curve elbow following Occam's razor.
• Never extrapolate polynomials beyond their training boundaries as higher powers rocket to infinity.