Principal Component Analysis (PCA)

Principal Component Analysis (PCA) is an unsupervised learning technique used to reduce the dimensionality of a dataset while preserving as much structure and information as possible.

Imagine you have a complex dataset with many features (dimensions). If you try to compress all those features down into just one or two, you inevitably lose some detail. PCA answers a critical question: Which direction should we project the data onto to lose the least amount of information?

Variance vs. Error

In the interactive lab above, we start with 2D data and try to compress it into a 1D line (an axis). As you rotate the axis, notice two metrics changing:

1. Variance (Captured): This measures how spread out the projected points are along the line. A high variance means the points haven't all clumped together; they still retain their distinct differences.
2. Orthogonal Error (Lost): This is the mean squared distance from the original 2D points to the 1D line. This distance represents the information lost in the compression.

Notice that the sum of these two values (Total Variance) is constant. Any variance captured by the projection reduces the orthogonal error. Therefore, maximizing the variance is mathematically identical to minimizing the information loss.

Finding the Principal Component

The direction that captures the absolute maximum variance—and therefore minimizes the orthogonal error—is called the First Principal Component (PC1).

If you click "Snap to PC1" in the lab, you will see the axis align perfectly with the longest spread of the data.

A Common Misconception

People often confuse PCA with Linear Regression because both draw a line through a scatter plot.

• Linear Regression predicts a target $y$ from a feature $x$. It draws a line that minimizes the vertical error (the distance straight up or down to the line).
• PCA does not predict anything; it finds the intrinsic structure. It minimizes the orthogonal error (the perpendicular distance to the line).

Try rotating the axis 90 degrees away from PC1. You will see the variance drop to its minimum and the error skyrocket to its maximum. This is the worst possible way to compress the data, as it discards the primary pattern entirely.