Dimensionality Reduction

One question sorts every method here: does it hand you a transform you can apply to a new row? PCA does. t-SNE never will, which is why it stays a plot.

PCA finds the cloud's long shared direction first, so two components hold 99.98 percent of the spread and the third column turns out never to have been independent.

Why Does This Exist?

TF-IDF twenty thousand support tickets and you get a matrix 38,000 columns wide. Nothing is wrong with it. It also cannot be plotted and makes anything distance-based unreliable, because at that width every pair of points sits at roughly the same distance apart (sparse data and high dimensionality).

Two reasons to want fewer columns. Preprocessing: fewer inputs to a model that runs next month, on rows nobody has collected yet. Looking: two axes on a screen, so you can see whether your classes separate at all.

One question sorts the methods: does it hand you a transform you can save and apply to a single new row. PCA does, LDA and TruncatedSVD do, t-SNE and UMAP do not. Pretending otherwise is the most common misuse here. This page covers PCA and its transform-giving relatives, the two that never give one, and the two setup mistakes that decide the result before any method runs.

Think of It Like This

A shadow on the wall and a seating plan

Hold a wire chair in front of a lamp. The shadow lost the depth, but the rule that made it is fixed: lamp there, wall here. Bring a second chair and its shadow appears under the same rule. That rule is a transform, the part you can ship.

Different job. Forty guests, and you shuffle the seating until friends sit close and strangers sit apart. The cliques are visible at a glance. Then guest forty-one turns up, and there is no rule to apply. The plan came out of the relationships among those forty. The gap between two tables measures nothing either.

Shadows are PCA. Seating plans are t-SNE.

Where it stops: a shadow is cast by physics you did not choose, while a projection is a choice of what to preserve. PCA keeps spread and discards the rest, so a direction with small variance but all the signal is gone for good.

How It Actually Works

PCA, and what maximum variance buys

Centre every column on its mean, then find the direction along which the projected points spread out most. That is the first principal component. The second is the direction of most remaining spread at a right angle to it. Those directions are the eigenvectors of the covariance matrix sorted by eigenvalue, equivalently the right singular vectors of the centred matrix (SVD). Each eigenvalue over their sum is that component's variance share, so you keep components until the total hits your target.

Neither input requirement is optional. Centre: variance is measured about the mean, and PCA has no other origin. Scale: variance carries units, so a column in centimetres outvotes the same column in metres 10,000 to one (feature scaling).

The others that hand you a transform

TruncatedSVD takes those same singular vectors without centring, which is why it runs straight on a sparse matrix. Centring would fill in every zero. It is what you use on TF-IDF, where it goes by latent semantic analysis.

LDA maximises between-class over within-class scatter instead of variance, so it is capped at one fewer component than you have classes. It reads the labels, so fitting it outside the fold leaks like a target encoder (cross-validation schemes).

Random projection multiplies by a random matrix and stops. Johnson and Lindenstrauss showed that is enough: 10,000 points with distances held within 50 percent need 442 dimensions, and ten times the points costs 110 more. Nothing is fitted, so nothing leaks. The learned non-linear version is an autoencoder bottleneck, at the price of a training run. That is the door into latent spaces and manifold learning.

The two that do not

t-SNE and UMAP build a neighbour graph, then arrange points in two dimensions so neighbours stay neighbours. Local structure survives. Global structure does not, and three things on those plots carry no information. Cluster size: a tight blob and a sprawling one can hold the same spread. Distance between clusters: opposite corners are not more different than touching blobs. Apparent gaps: that is where the layout ran out of pressure.

One knob changes the answer: perplexity in t-SNE, n-neighbors in UMAP. Turn it up and blobs merge, turn it down and one shatters into five. Diagnosis tools, and never a pipeline stage.

Worked example

Three columns on 300 people: height in centimetres, the same height in inches with a tenth of an inch of noise, and resting heart rate.

Standardise, run PCA, and the variance shares come out 66.81 percent, 33.17 percent and 0.01 percent. Two components hold 99.98 percent, because a third independent column was never there. Loadings on the first are 0.705, 0.705, 0.069: equal weight on the height pair, since $1/\sqrt{2} = 0.707$ , and heart rate barely registering.

Skip the standardising and those loadings become 0.929, 0.366, 0.056. Divide the first by the second: $0.929 / 0.366 = 2.54$ , the centimetres in an inch. Nothing about the data changed. The first component just described your ruler.

Code

1import numpy as np2
3rng = np.random.default_rng(0)4height_cm = rng.normal(170.0, 9.0, 300)5height_in = height_cm / 2.54 + rng.normal(0.0, 0.1, 300)  # same fact, different ruler6X = np.column_stack([height_cm, height_in, rng.normal(68.0, 7.0, 300)])7
8
9def pca(data: np.ndarray) -> tuple[np.ndarray, np.ndarray]:10    """Centre, take the singular values, square them. Loading signs are arbitrary."""11    _, s, vt = np.linalg.svd(data - data.mean(axis=0), full_matrices=False)12    return 100 * s**2 / (s**2).sum(), np.abs(vt[0])13
14share, pc1 = pca(X / X.std(axis=0))         # standardised, which is how to run it15print([round(float(v), 2) for v in share])  # -> [66.81, 33.17, 0.01]16print(np.round(pc1, 3))                     # -> [0.705 0.705 0.069]17print(np.round(pca(X)[1], 3))               # -> [0.929 0.366 0.056]

One expression separates those results: X / X.std(axis=0). Drop it and PC1 describes your unit rather than your data.

Watch Out For

Reading structure off a t-SNE or UMAP plot

Six well-separated blobs come back, two far apart on the right, and "segments 3 and 5 are the most distinct" goes into the summary. That claim was never in the data: positions came from a stochastic layout that only tried to keep near neighbours near, and dropping perplexity from 30 to 5 turns six blobs into fifteen.

The worse version happens next. You keep the two output columns and hand them to a classifier. There is no transform for a new row, so it cannot be served, and the layout saw every row you own, so the score was inflated before the classifier started.

Symptom: a two-column feature with no transform method and suspiciously clean clusters. Fix: want two columns feeding a model, use PCA or TruncatedSVD, inside the fold.

PCA on unscaled columns, fitted before the split

Both bugs come out of PCA().fit(X) on the raw frame.

The scaling half is quiet. Annual income in the tens of thousands next to tenure in single digits means income's variance is eight orders of magnitude larger, so PC1 is income with a rounding error attached and "99 percent of the variance explained" describes currency.

The leakage half is louder. PCA learns column means and a rotation from whatever rows you show it, so fitting on the full matrix hands your test set a projection built partly out of itself. Validation improves, production does not, nothing errors.

Symptom: PC1 loadings that mirror the raw unit sizes, or validation that beats production. Fix: standardise, and put both steps in a pipeline so every fold refits (data leakage).

The Quick Version

Does the method give you a transform you can apply to a new row. PCA, LDA and TruncatedSVD do. t-SNE and UMAP do not.
PCA takes orthogonal directions of maximum variance, sorted by explained variance, and wants centred, usually scaled input, because variance carries units.
TruncatedSVD is PCA without centring, which is what makes it safe on sparse input. LDA is supervised, capped at one fewer component than you have classes, and leaks outside the fold.
Cluster size, between-cluster distance and empty gaps on a t-SNE plot mean nothing, and the neighbourhood knob changes the picture completely.
Two of three columns being one height in two units gives shares of 66.81, 33.17 and 0.01 percent. Unscaled, PC1 loadings sit in a 2.54 ratio: the unit conversion.