PCA Derivation

Principal Component Analysis rotates our coordinate system so that the first axis points in the direction where the data varies the most.

Finding the principal component axis that maximizes the data's variance.

Why Does This Exist?

High-dimensional data contains a lot of redundant information and noise. PCA allows us to compress the data into fewer dimensions while retaining as much of the original variance (the "signal") as mathematically possible.

Think of It Like This

Imagine shining a flashlight on a 3D object to project its shadow onto a 2D wall. Depending on how you rotate the object, the shadow might look like a thin line or a broad shape. PCA is the algorithm that perfectly rotates the object so its 2D shadow captures the maximum possible spread and detail.

How It Actually Works

The derivation of PCA is rooted in linear algebra and can be shown via eigen-decomposition:

Center the Data: Subtract the mean from each feature so the dataset $X$ is centered at the origin.
Covariance Matrix: Compute the empirical covariance matrix $C = \frac{1}{n-1} X^T X$ . This matrix describes how every pair of features varies together.
Maximize Variance: We want a unit vector $w$ such that the variance of the projected data, $w^T C w$ , is maximized.
Lagrange Multiplier: Using the constraint $w^T w = 1$ , we formulate the Lagrangian $L = w^T C w - \lambda(w^T w - 1)$ .
Eigenvalue Problem: Taking the derivative with respect to $w$ and setting it to zero gives $C w = \lambda w$ . The vector $w$ that maximizes the variance is exactly the eigenvector of $C$ corresponding to its largest eigenvalue $\lambda$ .

Code

1import numpy as np2
3def get_principal_component(X: np.ndarray) -> np.ndarray:4    # X is shape (n_samples, n_features)5    X_centered = X - np.mean(X, axis=0)6    covariance_matrix = np.cov(X_centered, rowvar=False)7    8    eigenvalues, eigenvectors = np.linalg.eigh(covariance_matrix)9    10    # eigh returns eigenvalues in ascending order, take the last one11    return eigenvectors[:, -1]12
13data = np.array([[1, 2], [3, 4], [5, 6]])14print(np.round(get_principal_component(data), 3))15# -> [0.707 0.707]

Watch Out For

Scale sensitivity: Because PCA maximizes absolute variance, features with larger numeric scales will completely dominate the principal components. You must standardize your features (e.g., zero mean, unit variance) before applying PCA.

The Quick Version

PCA reduces dimensionality by projecting data onto orthogonal axes of maximum variance.
The principal components are the eigenvectors of the data's covariance matrix.
The amount of variance explained by each component is proportional to its eigenvalue.
It can also be efficiently computed using Singular Value Decomposition (SVD) on the centered data matrix.