Marchenko-Pastur Law

The exact shape noise paints on sample eigenvalues. Given the ratio of features to samples, it draws the bulk, the edges and the mean in one curve.

At ratio one half the noise bulk runs from 0.09 to 2.91 with mean 1.0, so only spikes past 2.91 count as signal

Why Does This Exist?

The sample covariance $S = XX^T/n$ is what PCA actually diagonalises, where $X$ is the $d \times n$ data matrix and $n$ is the sample count. When $d = 500$ and $n = 1000$ , $S$ is $500 \times 500$ of pure sampling noise even if the truth is identity. Classical asymptotics with fixed $d$ say $S$ approaches truth, but here $d/n = 0.5$ stays put and the spread never vanishes.

The Marchenko-Pastur law is the exact limit for that regime. It takes only the ratio $y = d/n$ and returns the full noise histogram: where it starts, where it ends, how it curves and where its mean sits. With that curve drawn, factor selection becomes visual: bulk is noise, spikes past the right edge are candidates. Random matrix theory frames both classical shapes; this page is the covariance one, while Wigner semicircle law handles symmetric noise.

Prerequisite in one paragraph. You need the empirical spectral distribution, the histogram of the $d$ sample eigenvalues, and the ratio $y = d/n$ . Population eigenvalues describe truth, sample eigenvalues describe the noisy estimate. Background is in eigenvalues and eigenvectors and singular value decomposition.

Think of It Like This

A pond ripple photographed with few pixels

Photograph a smooth ripple with a coarse sensor and each pixel averages its patch. The photo shows lumps the water never had, and the lump pattern depends only on the pixel-to-wavelength ratio, not on the day. Finer sensor, same ratio, same lump texture.

Marchenko-Pastur is that texture chart for covariances. The ratio $d/n$ is the coarseness. It predicts the lump pattern pure noise produces, so a real wave standing above the texture must be signal.

Where the analogy stops: pixels blur locally while covariance noise couples globally through the eigenbasis. The predicted curve is also exact only in the large-size limit, though it is already close at $d = 500$ .

How It Actually Works

Let $X$ have independent entries with mean zero and variance $1$ , and form $S = XX^T/n$ . As $d, n \to \infty$ with $d/n \to y$ in $(0, 1)$ , the eigenvalue histogram of $S$ converges to the density $f(x) = \sqrt{(b-x)(x-a)}/(2\pi y x)$ on $[a, b]$ with $a = (1-\sqrt{y})^2$ and $b = (1+\sqrt{y})^2$ . Here $a$ is the left edge, $b$ the right edge, and the mean of the distribution stays $1$ .

For $y = 0.5$ : $\sqrt{y} \approx 0.7071$ , $a = (0.2929)^2 \approx 0.0858$ , $b = (1.7071)^2 \approx 2.914$ . The bulk spans $0.09$ to $2.91$ with mean $1$ . The density peaks left of centre and falls to zero as a square root at each edge, which is why histograms look cliff-sided on the right. When $y > 1$ a point mass at zero appears for the $d - n$ null directions; the formula extends with that atom.

Worked example

Take $d = 500$ noise features and $n = 1000$ samples, $y = 0.5$ . Theory predicts about $68$ percent of the $500$ eigenvalues below $1.0$ and the largest near $2.86$ in a typical draw, matching the diagram. The average of all $500$ stays $1.0$ to two decimals because trace is preserved: noise redistributes variance, it does not create it.

Now add one factor: population eigenvalues $4, 1, 1, \ldots$ . The spike maps to about $4.6$ past the $2.91$ edge and separates cleanly. Repeat with spike $1.8$ : it lands near $2.1$ inside the bulk and vanishes. The edge $b = 2.91$ is the practical detection line at this ratio, and it slides with $y$ : at $y = 0.1$ it is only $1.73$ , at $y = 0.9$ it reaches $3.80$ .

Watch Out For

Using the y equals 0 limit on high-ratio data

As $y \to 0$ the law collapses to a point at $1$ , the classical fixed- $d$ picture. Applying that point-mass thinking at $y = 0.5$ declares every eigenvalue above $1.2$ a factor. Roughly a third of pure noise sits there, so the factor count explodes.

Fix: compute $y = d/n$ first and draw the bulk for that $y$ . Never compare high-dimensional eigenvalues against $1.0$ plus a fudge.

Forgetting the point mass when features exceed samples

When $d > n$ , $S$ has at least $d - n$ exact zeros. The continuous density covers only the $n$ nonzero eigenvalues. Normalising the histogram over all $d$ without the atom at zero shifts every percentile and hides the true right edge.

Fix: separate the zeros first. Fit the bulk to the $n$ nonzero values with $y = d/n > 1$ form including the atom weight $1 - 1/y$ at zero.

The Quick Version

Marchenko-Pastur maps ratio $y = d/n$ to the full noise eigenvalue histogram with edges $(1 \pm \sqrt{y})^2$ .
At $y = 0.5$ the bulk runs 0.09 to 2.91 with mean 1.0 and a right cliff edge.
Only spikes past the right edge count as detectable signal at that ratio.
Trace is preserved: noise spreads variance without creating it.
Above ratio $1$ , remember the atom of exact zeros before fitting the bulk.