t-SNE

It keeps neighbours together and protects nothing else, so the gaps between the clusters you can see, and the sizes of those clusters, carry no information.

Why Does This Exist?

You have a thousand bird recordings and a model that turns each into 128 numbers. One question: do the three species separate, or does your model hear them as the same thing?

The obvious move is PCA. First two components, plot, look. Often you get one grey blob holding 22 percent of the variance, and you cannot tell whether the structure is not there or whether PCA could not reach it. It only rotates, and three species on a curved sheet in 128 dimensions do not flatten into separate patches under any rotation.

That is the wall t-SNE was built against. It gives up on geometry entirely and optimises for one thing: whoever was your neighbour before should be your neighbour in the picture. Everything else is a by-product. For the alternative that does embed new points, read UMAP next, and for scoring separation with real numbers, clustering evaluation.

Think of It Like This

The subway map that lies about distance

A subway map draws adjacent stations evenly spaced whether they are 300 metres apart or three kilometres, inflates the central zone so labels fit, and squeezes the outer branches. Nobody complains: the map only ever promised which stations connect to which, and in what order.

That is t-SNE. Neighbourhoods are the promise; distance and area were sacrificed so they could fit on a flat page at all.

The mistake is bringing a ruler. Two lines at opposite edges of the sheet can run parallel down the same street. The map did not lie; you asked a question it never claimed to answer.

And when a new station opens, the designer redraws the map; nothing places one stop without moving the rest. Same gap in t-SNE, and it is why the thing cannot sit in a pipeline.

How It Actually Works

From distances to neighbour probabilities

Take one recording. Measure its distance to all 999 others, then turn those distances into probabilities: if this call picked a neighbour at random in proportion to a bell curve centred on itself, how likely is each of the others? Close calls take almost all the probability; distant ones almost none.

The width of that bell curve is set per recording, by bisection, so the effective number of neighbours matches a number you choose: perplexity. Perplexity 30 means roughly "treat about 30 recordings as this one's local circle".

A concrete bandwidth

One call sits at squared distances $[1, 4, 9, 36]$ from four others. At width $\sigma = 1$ the neighbour distribution is $[0.8055, 0.1797, 0.0148, 0.0]$ , effectively 1.724 neighbours. Widen to $\sigma = 2$ and it softens to $[0.4836, 0.3324, 0.1779, 0.0061]$ , or 2.874 neighbours. At $\sigma = 4$ it spreads to $[0.3307, 0.3011, 0.2575, 0.1108]$ , or 3.745. Perplexity does not select neighbours; it selects the bandwidth, and the snippet below prints these rows.

The heavy tail, and what the objective actually pays for

Similarity in the map comes from a t-distribution with one degree of freedom, not a bell curve. Its tail is fat: at four times the distance it still assigns real probability where a bell curve has effectively none.

That fixes a counting problem. In 128 dimensions a point can have far more roughly-equidistant neighbours than a flat page has room for, so a faithful map jams everything into the middle. The heavy tail lets moderately distant pairs sit far apart without the objective charging much. All that white space between groups? That is the tail, not the data.

Gradient descent then moves the 2D points to shrink the KL divergence between the two sets of probabilities, one number for how badly one distribution stands in for another (the details):

D_{KL}(P \parallel Q) = \sum_{i \neq j} p_{ij} \log \frac{p_{ij}}{q_{ij}}

Here $p_{ij}$ is a pair's neighbour probability in the original 128 dimensions, $q_{ij}$ is the same pair's probability in the map, and the sum runs over every pair. The asymmetry is the whole story. True neighbours drawn far apart carry a big penalty; strangers drawn close together carry almost none. So neighbourhoods get protected fiercely, and everything else is slack.

What the picture does not tell you

Distances between clusters mean nothing. The objective only pays for getting neighbourhoods wrong, so two groups at opposite ends of the map may be closer in the original space than two drawn side by side. The diagram shows that swap.

Cluster sizes mean nothing either. Each point's bell curve scales to its own local density, expanding sparse regions and contracting dense ones. A tight cluster of near-identical calls can come out a broad loose cloud.

Perplexity changes the picture, not the data. At 5 you get a scatter of small islands; at 100 they merge into continents. Neither is the truth. Trust only structure that survives several values.

Two runs disagree. Random initialisation and a stochastic objective give a different map each time, rotated, rearranged, sometimes with one group split in two. Fix the seed for reproducibility; that is not stability.

Code

Perplexity does not select neighbours. It selects a bandwidth per point, and here is what three bandwidths do to one call's distribution over four fixed distances.

1import math2
3
4d2 = [1.0, 4.0, 9.0, 36.0]  # squared distances from one call to four others5
6
7def row(sigma: float) -> list[float]:8    w = [math.exp(-d / (2 * sigma * sigma)) for d in d2]9    total = sum(w)10    return [v / total for v in w]  # one call's distribution over the others11
12
13def perplexity(p: list[float]) -> float:14    ent = -sum(v * math.log2(v) for v in p if v > 0)15    return 2.0 ** ent16
17
18for sigma in (1.0, 2.0, 4.0):19    p = row(sigma)20    print(sigma, [round(v, 4) for v in p], round(perplexity(p), 3))21# -> 1.0 [0.8055, 0.1797, 0.0148, 0.0] 1.72422# -> 2.0 [0.4836, 0.3324, 0.1779, 0.0061] 2.87423# -> 4.0 [0.3307, 0.3011, 0.2575, 0.1108] 3.745

Wider bell, softer distribution, larger effective neighbourhood. That number fixes what "local" means before a single point moves.

Watch Out For

Quoting a distance or a size off the plot

Someone puts a t-SNE map on a slide and writes underneath that two species sit "roughly three times the within-group spread apart", or that one group is more varied because its blob is bigger. Both numbers came off the screen with a ruler; change the seed and both change.

State separation with a number computed in the original 128 dimensions, a silhouette score or nearest-neighbour purity; clustering evaluation covers the options. Use the plot for what it is good at: noticing you have three groups rather than two, or one recording sitting deep inside the wrong species.

Using it as a preprocessing step

The pattern: 128 dimensions down to 2, train a classifier on the two columns, get a strong cross-validated score, then find at deployment there is no way to place tomorrow's recording. There is no transform, a new point arrives only by refitting, and refitting moves every other point.

The score was not real either. The embedding saw all 1,000 recordings including the ones you scored on, so held-out rows helped decide where they themselves would land, leakage, in the flattering direction as usual. Need low-dimensional features inside a model? PCA and UMAP both embed new points from an already-fitted model.

The Quick Version

Neighbour probabilities in the original space, the same in 2D with a heavy-tailed t-distribution, then minimise the KL divergence between them.
The heavy tail stops everything crowding into the centre; it is where the white space between groups comes from.
Neighbourhoods are preserved. Between-cluster distances, cluster sizes and overall geometry are not.
Perplexity sets neighbourhood size. Check several; structure that shows at only one value is not structure.
Non-deterministic by construction. Two runs, two different pictures of identical data.
No transform for new points, so it belongs on a slide, not in a pipeline.