Linear Regression

Prediction is a weighted sum plus an intercept, fitted to make the leftover errors smallest. Those leftovers are the most useful thing the model hands back.

Three points at (1,2), (2,3) and (3,5) fit the line y equals 1.5x plus 0.33 with residuals 0.17, minus 0.33 and 0.17 that square and sum to 0.167

Why Does This Exist?

A fleet operator prices three hundred vans before auction. The odometer explains most of the price, so plot price against kilometres for vans that already sold, draw the best line, and read the next van off it.

That is linear regression, still the first model for a numeric target. Not because it wins on accuracy, but because it tells you what stronger models hide. One weight per feature. Leftover errors you can plot. Failure in a shape you can name.

One idea underneath it. A weighted sum multiplies each number about a van by its own weight and adds them up. That is the dot product written longhand. Add a constant intercept and you have every prediction on this page.

This page covers the fit, the residuals, and the traps. The geometry behind the closed form lives in ordinary least squares. How to score the fit lives in regression metrics. What to do when features correlate lives in ridge regression.

Think of It Like This

A ruler held by vertical springs

Print sold vans as dots, price up and kilometres across. Lay a steel ruler anywhere over the cloud. Hook one spring per dot from the dot straight up or down to the ruler, vertical only, never diagonal, then let go.

The ruler slides and tilts until it rests. That resting place is the least squares fit. A spring stores energy as the square of its stretch, so resting minimises the sum of squared vertical gaps, which is the definition of the fit.

Each dot is one van, the ruler is the prediction line, each spring length is one residual, and the square in the spring energy is the square in least squares.

Where it stops: springs explain vertical errors only. They say nothing about a curved cloud, and one overstretched spring pulls nine times as hard for three times the stretch, so a single odd sale can drag the whole ruler.

How It Actually Works

Stack vans into a matrix $X$ , one row per van, one column per feature, plus a leading column of ones for the intercept. Put prices in a vector $y$ . Every prediction at once is $X w$ , where $w$ is one weight per column. The fit picks the $w$ with smallest total squared error:

w = (X^\top X)^{-1} X^\top y

Here $X^\top$ is $X$ with rows and columns swapped, so $X^\top X$ is square with one row per feature. No learning rate and no seed. One answer when the product inverts. With fifty thousand sparse columns that matrix is too large to build, which is where gradient descent takes over.

Worked example with three points

Take points $(1, 2)$ , $(2, 3)$ , $(3, 5)$ . Mean of $x$ is $2$ . Mean of $y$ is $10 / 3 \approx 3.33$ .

Slope is covariance over variance. Deviations in $x$ are $[-1, 0, 1]$ . Deviations in $y$ are $[-1.33, -0.33, 1.67]$ . Numerator is $(-1)(-1.33) + 0 + (1)(1.67) = 3.0$ . Denominator is $1 + 0 + 1 = 2$ . Slope is $3.0 / 2 = 1.5$ . Intercept is $3.33 - 1.5 \times 2 = 0.33$ .

Predictions are $1.5 \times 1 + 0.33 = 1.83$ , $1.5 \times 2 + 0.33 = 3.33$ , $1.5 \times 3 + 0.33 = 4.83$ . Residuals, actual minus predicted, are $2 - 1.83 = 0.17$ , $3 - 3.33 = -0.33$ , $5 - 4.83 = 0.17$ . Squares are $0.029$ , $0.109$ , $0.029$ . Sum is $0.167$ . That sum is the diagram headline: three dots, one line, three vertical gaps that square to $0.167$ .

Residuals are the diagnostic

A residual is one van minus its prediction, one spring length. Plot residuals against predicted price. Structureless fog is what you want. A smile or frown means a line on curved data. Vans lose value fastest early, so a straight line overprices the middle and underprices the ends.

Correlated features destabilise weights

Odometer and age correlate near $0.9$ in a real fleet. Include both and coefficients swing large, and refitting on another year can flip a sign. Predictions stay fine.

Two near duplicate columns mean many weight pairs give almost the same predictions, so the fit has no preference and picks on noise. Need stable weights for pricing rules. Drop one column or shrink with ridge regression.

Code

Fit the three point example with plain Python and confirm $0.167$ .

1xs: list[float] = [1.0, 2.0, 3.0]2ys: list[float] = [2.0, 3.0, 5.0]3
4mean_x: float = sum(xs) / len(xs)5mean_y: float = sum(ys) / len(ys)6num: float = sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys))7den: float = sum((x - mean_x) ** 2 for x in xs)8slope: float = num / den9intercept: float = mean_y - slope * mean_x10
11resids: list[float] = [y - (slope * x + intercept) for x, y in zip(xs, ys)]12sse: float = sum(r * r for r in resids)13
14print(f"{slope:.2f} {intercept:.2f}")  # -> 1.50 0.3315print(f"{resids[0]:.2f} {resids[1]:.2f} {resids[2]:.2f}")  # -> 0.17 -0.33 0.1716print(f"{sse:.3f}")  # -> 0.167

The printout matches the hand arithmetic. Change the third point from $5.0$ to $8.0$ and watch the slope jump past $2.0$ . That sensitivity is the spring picture in numbers.

Watch Out For

Reading a coefficient as a cause

Service visits get a positive weight, so each extra visit predicts a higher price. A report claims servicing raises resale by $400 a visit.

Symptom: a decision about spending follows from a weight alone. Fix: remember the weight describes vans that sold, holding other columns fixed. Vans worth keeping got serviced. Direction needs an experiment, not a regression. Never price an action off a weight.

Trusting R-squared to approve the model

R-squared rises with every added column, even a column of noise. A report at $0.94$ with twenty features says nothing about feature twenty one.

Symptom: training score climbs while held out error in currency stalls near $2,400 per van. Fix: judge on held out error in units you can price, as in regression metrics. A model at $0.88$ off by $900 beats one at $0.94$ off by $2,400.

The Quick Version

Linear regression predicts a weighted sum plus an intercept fitted by least squares.
Our three points fit y equals 1.5x plus 0.33 with residuals 0.17, minus 0.33 and 0.17.
Those three squared gaps sum to 0.167, the score the line minimises.
Residuals plotted against fitted values reveal wrong form as visible shape.
Correlated inputs leave predictions fine while coefficients swing wildly.
R-squared only rises with new columns, so judge models on held out error in real units.