Principal Component Analysis

Turn the axes until the first runs along the widest spread, then drop the flat directions that carry almost nothing.

Two eigenvalues 8 and 2 sum to 10, so the long axis keeps 0.80 of the spread and the short axis keeps 0.20

Why Does This Exist?

An apple line records twelve numbers per fruit: weight, diameter, height, firmness, sugar, three colours, blemish counts. Weight and diameter move together because wide apples weigh more. Twelve columns hold perhaps five ideas, and the repeats waste plots, distances, and model capacity.

PCA counts the real ideas and returns new columns that never repeat each other. It is one method for fewer better columns. The general case for why fewer columns help lives in the data section treatment of the same problem, and the nonlinear picture that follows two flat components lives in t-sne with its sequel in umap.

Two ideas help first. Variance along a direction means spread measured along it. An eigenvector of the covariance matrix means a direction the matrix only stretches, with the stretch factor called its eigenvalue. The computation underneath uses the decomposition in singular value decomposition.

Think of It Like This

The biggest shadow a hammer can cast

Hold a claw hammer under a bright lamp. Handle pointed at the wall gives a stubby blob. Turn it side on and the shadow stretches long, showing handle plus claw. That long direction keeps the most shape once flattened.

The long shadow is the first component, the most spread kept. The width across the head is the second, the most left at right angles. Thickness into the wall never reaches any shadow.

Where it stops: no shadow shows thickness, and no choice of lamp angle recovers it. Dropped components are gone the same way, and centred data is required or the shadow position reports where the hammer sits instead of its shape.

How It Actually Works

Centre each column by subtracting its mean, so the cloud sits at the origin. Form the covariance matrix, meaning the table of how columns vary together. Its eigenvectors are the new axes. Its eigenvalues are the variances along them. Order largest first. The explained share of component $i$ , meaning the fraction of total spread it carries, is:

\text{ratio}_i = \frac{\lambda_i}{\sum_j \lambda_j}

Here $\lambda_i$ means the eigenvalue for component $i$ , the variance along that axis, and the sum runs over all components. In practice the axes come from the singular value decomposition of the centred matrix, which avoids squaring the data and keeps the arithmetic stable.

Worked example: eigenvalues 8 and 2

Suppose two centred columns have covariance matrix with 5 on the diagonal and 3 off diagonal. The trace is $5 + 5 = 10$ . The determinant is $25 - 9 = 16$ . For a two by two matrix the eigenvalues solve to:

\lambda = \frac{10 \pm \sqrt{100 - 64}}{2} = \frac{10 \pm 6}{2}

So $\lambda_1 = 8$ and $\lambda_2 = 2$ . The total is $8 + 2 = 10$ . The shares are:

8 / 10 = 0.80, \qquad 2 / 10 = 0.20

The long axis keeps 80 percent of the spread and the short axis keeps 20 percent. Drop the short axis and the cost is exactly the 0.20 slice it carried. That is the diagram: one long mint axis labelled 0.80, one short grey axis labelled 0.20, one point plus its square drop onto the long axis.

Two steps you cannot skip

Centre the columns. Skip it and the first axis points at the average apple instead of the spread direction. Standardize when units differ. Variance carries units, so grams with variance in the hundreds outvote sugar with variance under one for no reason except small units. Divide by the scale first or the units decide the answer.

What it is not

A rotation plus deletion, nothing more. Linear structure survives, curves do not. Keep every component and the rotation reverses exactly. It is also unsupervised: the target never enters the computation, so a thin predictive direction can sit in the dropped slice. Each component mixes every original column, so readable names past the first are usually fiction. Report the loadings instead.

Code

Rebuild the two by two split from its eigenvalues with plain Python.

1eig: list[float] = [8.0, 2.0]2total: float = sum(eig)3ratios: list[float] = [v / total for v in eig]4
5print(f"{total:.1f}")  # -> 10.06print([f"{r:.2f}" for r in ratios])  # -> ['0.80', '0.20']7print(f"{1.0 - ratios[0]:.2f}")  # -> 0.20

Total prints 10.0, shares print 0.80 and 0.20, and the drop cost prints 0.20, matching the hand division. Change 3 off diagonal to 4 and watch the split sharpen toward 0.90 and 0.10.

Watch Out For

Keeping axes by variance before a classifier

Symptom: accuracy falls after keeping the axes that cover 95 percent of variance, then climbs as thin axes return, including ones near half a percent each. Fix: variance never promised relevance because the target never entered the rotation. Choose the count by validation score, keep the rotation for compression and plots, and switch to a supervised method when prediction is the goal.

Fitting the rotation on every row then scoring those rows

Symptom: cross validated scores look strong and production scores lower, with no model change to blame. Fix: fit means, scales, and axes on training rows only, then transform the rest. Inside cross validation the whole PCA step sits in the pipeline and refits per fold. See the general leakage pattern before trusting the first number.

The Quick Version

PCA returns eigenvectors of the covariance of centred data, ordered by eigenvalue from large to small.
Explained share is one eigenvalue over the sum of all eigenvalues.
Eigenvalues 8 and 2 sum to 10, so the shares are 0.80 and 0.20 and dropping the short axis costs 0.20.
Centre always, standardize whenever columns carry different units.
Computation runs through SVD on the centred matrix, never by squaring into a covariance first.
The method is linear and unsupervised, so curves do not survive and thin predictive directions can land in the dropped slice.