Principal Component Analysis (PCA)

Principal Component Analysis (PCA) is an unsupervised learning technique used to reduce the dimensionality of a dataset while preserving as much structure and information as possible.

Imagine you have a complex dataset with many features (dimensions). If you try to compress all those features down into just one or two, you inevitably lose some detail. PCA answers a critical question: Which direction should we project the data onto to lose the least amount of information?

Variance vs. Error

In the interactive lab above, we start with 2D data and try to compress it into a 1D line (an axis). As you rotate the axis, notice two metrics changing:

1. Variance Kept: how spread out the projected points are along the line. High variance means the points have not all clumped together; they still retain their distinct differences.
2. Variance Lost: the mean squared perpendicular distance from the original 2D points to the 1D line. That distance is the information the compression discards.

The two always sum to the same number. On this dataset the total is 7.802, and it stays 7.802 at every angle you can reach, because the total spread of a cloud about its own centre is a property of the cloud and not of the line you choose to describe it with. So any variance the projection captures is variance the error no longer holds. Maximising the one is not a competing goal with minimising the other; it is the same goal stated twice.

Why the percentage, not the variance

The lab also reports an Explained percentage, which is just the variance kept divided by that total. It is worth having on screen because 7.523 on its own tells you nothing — a variance is in squared feature units, so whether it is large depends on units you happen to have chosen. Divided through, it becomes "this axis keeps 96.4% of what was there", which means the same thing on any dataset. This is the number real implementations report as explained_variance_ratio_.

Finding the Principal Component

The direction that captures the absolute maximum variance — and therefore minimises the error — is the First Principal Component (PC1). For two features it has a closed form; there is no iteration and no tolerance to tune.

Click Snap to PC1 in the lab and the axis jumps to 43.3°, where it keeps 96.4% of the variance and leaves an orthogonal error of 0.279. You can also hunt for it by hand: the marker on the slider shows where it is, and the challenge accepts anything within about three degrees, which is a quarter of a percentage point of explained variance.

Centre first

Every function behind this lab takes the centroid as an argument, and the axis is drawn pivoting through it. That is not a presentational choice. Variance is defined about the mean, so if you skip the centring the first component points from the origin toward the cloud rather than along its spread — you would recover where the data sits, not how it varies.

A Common Misconception

People often confuse PCA with Linear Regression because both draw a line through a scatter plot.

• Linear Regression predicts a target $y$ from a feature $x$. It draws the line that minimises the vertical error — the distance straight up or down to the line.
• PCA does not predict anything; it finds the intrinsic structure. It minimises the orthogonal error, the perpendicular distance, and treats both features symmetrically.

The dashed drops in the lab are perpendicular to the axis, not vertical, and that is the whole difference. The two lines coincide only when the data is perfectly correlated.

Breaking it on purpose

Turn the axis to -46.7°, a quarter circle away from PC1. Nothing errors. Every point still gets a number. But the split inverts exactly: variance kept falls to 0.279 and variance lost rises to 7.523 — the optimum's own two figures with their roles exchanged — so the axis now explains 3.6% instead of 96.4%. Look at the 1D strip and the eleven points are almost on top of one another. This is the worst available way to compress the data, and the only thing that tells you so is the number.

One more caveat

This lab does not standardise its features, and it does not need to: both are on the same scale already. Real data usually is not. PCA maximises variance in whatever units it is given, so re-expressing one feature in millimetres instead of metres multiplies its variance by a million and drags the first component onto it. When features are on unlike scales, standardise before you decompose.