PCA Derivation

Center the data, then point the first axis where the spread is largest. Those best axes are the covariance eigenvectors, ranked by their eigenvalues.

Centered data with covariance 5 3 3 5 stretches most along the 0.707 diagonal, where variance 8 beats variance 2, so keeping that axis preserves 80 percent of the spread.

Why Does This Exist?

A table with 200 correlated columns will not fit in memory comfortably, will overfit a small sample, and will not plot. The columns move together, so the true spread lives in far fewer directions than the column count suggests. The fix is a rotation: keep the few directions that carry most of the variance and drop the rest, with a number attached to how much was kept.

This page derives that rotation. It shows why the best directions are exactly the eigenvectors of the covariance matrix and why the eigenvalues are the variances along them. The directions themselves belong to eigenvalues and eigenvectors; the symmetric-matrix machinery is in eigendecomposition of symmetric matrices; the stable way to compute them without forming the covariance is singular value decomposition.

The one idea you need first

Variance measures spread along one axis: the average squared distance from the mean. Covariance measures joint spread: for centered data $X$ with $n$ rows, $C = X^T X / (n - 1)$ holds a variance on each diagonal entry and a co-movement on each off-diagonal. Projecting onto a unit vector $w$ gives scores $Xw$ , whose variance is $w^T C w$ . PCA is the search for the $w$ that makes that number largest.

Think of It Like This

Finding the long axis of a scattered deck of cards

Scatter a deck of cards on a table so most lie roughly along one diagonal with a little sideways jitter. Ask which straight line captures the layout best, and you point along the diagonal: cards spread far along it and barely at all across it.

That diagonal is the first principal component. Its length of spread is the first eigenvalue, here 8 against 2 across, so 80 percent of the layout lives along one line. The second axis must sit at a right angle to the first, capturing whatever spread remains.

Where it stops: cards have a visible long side, while data axes must be computed. The analogy also hides centering: slide the whole scatter without re-aiming and the best line follows the slide unless you subtract the mean first.

How It Actually Works

Center the $n \times d$ data $X$ so each column has mean zero. The empirical covariance $C = X^T X / (n - 1)$ is a $d \times d$ symmetric matrix. For a unit direction $w$ with $w^T w = 1$ , the variance of the projected scores is $w^T C w$ . In plain words: $w$ is the candidate axis, $C$ describes the cloud, and the quadratic form reads off the spread along $w$ .

The eigenvalue step

Maximize $w^T C w$ subject to $w^T w = 1$ with a Lagrange multiplier $\lambda$ :

L = w^T C w - \lambda (w^T w - 1)

Differentiating and setting the gradient to zero gives $C w = \lambda w$ . So the maximizer is an eigenvector of $C$ , and left-multiplying by $w^T$ shows the value there is $\lambda$ itself: the eigenvalue is the projected variance. The full symmetric case is in eigendecomposition of symmetric matrices, and orthogonality and projections explains why the next axes can be chosen perpendicular without losing anything.

Worked example

Take a centered cloud whose covariance is already accumulated as:

C = \begin{bmatrix} 5 & 3 \\ 3 & 5 \end{bmatrix}

In plain words: each variable has variance 5, and the off-diagonal 3 says they move together. The characteristic equation is $(5 - \lambda)^2 - 9 = 0$ , which expands to $\lambda^2 - 10\lambda + 16 = 0$ and factors as $(\lambda - 8)(\lambda - 2) = 0$ .

So $\lambda_1 = 8$ with eigenvector $v_1 = [1, 1] / \sqrt{2} \approx [0.707, 0.707]$ , and $\lambda_2 = 2$ with $v_2 = [1, -1] / \sqrt{2} \approx [0.707, -0.707]$ . Total variance is $8 + 2 = 10$ , so the first axis explains $8/10 = 80$ percent and the second $20$ percent. Keeping only PC1 drops from two numbers per point to one while preserving 80 percent of the spread, which is the diagram above.

A point $x = [2, 0]$ scores $z_1 = x \cdot v_1 = 2 \times 0.707 + 0 = 1.414$ on PC1. Reconstructing from PC1 alone gives $1.414 \times [0.707, 0.707] \approx [1, 1]$ , so the residual $[1, -1]$ is exactly along PC2, the direction we chose to discard.

Code

Pure standard library, no extra packages. It solves the same 2-by-2 by trace and determinant, then reproduces the 80 percent figure and the projection.

1import math2
3# Covariance from the worked example.4a, b, c, d = 5.0, 3.0, 3.0, 5.05trace = a + d            # 10.06det = a * d - b * c      # 16.07disc = math.sqrt(trace * trace - 4 * det)  # sqrt(36) = 6.08lam1 = (trace + disc) / 2  # 8.09lam2 = (trace - disc) / 2  # 2.010print(lam1, lam2)        # -> 8.0 2.011print(round(lam1 / (lam1 + lam2), 4))  # -> 0.812
13v1 = [1 / math.sqrt(2), 1 / math.sqrt(2)]14x = [2.0, 0.0]15z1 = x[0] * v1[0] + x[1] * v1[1]16print(round(z1, 4))      # -> 1.414217recon = [z1 * v1[0], z1 * v1[1]]18print([round(v, 4) for v in recon])  # -> [1.0, 1.0]

The prints give 8.0 and 2.0, an 0.8 explained ratio, a score of 1.4142, and a reconstruction of [1.0, 1.0], matching the hand computation.

Watch Out For

Forgetting to center before forming the covariance

The symptom is a first component that points at the mean instead of the spread: with all values near 1000, the uncentered second moment is dominated by $1000^2$ and PC1 becomes roughly the all-ones direction. Explained variance looks near 100 percent while downstream regression barely improves. Subtract the column means first, then form $X^T X / (n - 1)$ , and check that each column averages to zero before decomposing.

Letting one raw scale dominate the variance

The symptom is a PC1 that is essentially one column: income in units against age in years gives income a variance millions of times larger, so the top eigenvector loads almost entirely on income. The fix is to standardize each column to unit variance when units differ, which replaces covariance with correlation. Skip this only when all columns share units and the raw scales carry meaning, such as pixels from one camera.

The Quick Version

PCA keeps the directions of largest projected variance and ranks them by eigenvalue.
Center first, form $C = X^T X / (n - 1)$ , then solve $C v = \lambda v$ for axes $v$ and variances $\lambda$ .
With covariance 5 3 3 5, variances are 8 and 2, so PC1 keeps 80 percent of the spread.
The PC1 axis is $[0.707, 0.707]$ ; a point $[2, 0]$ scores 1.414 and rebuilds as $[1, 1]$ .
Standardize columns with different units, or the largest raw scale wins every component.
Computation at scale uses SVD on centered $X$ instead of forming $C$ .