Skip to content
AI360Xpert

t-SNE & UMAP Math Basics

These algorithms compress complex high-dimensional data into 2D or 3D maps by ensuring that points close together in high dimensions stay close together in low dimensions.

Preserving local neighborhood clusters during dimensional projection.
Preserving local neighborhood clusters during dimensional projection.

Why Does This Exist?

Linear methods like PCA are great for global structure but terrible at preserving local clusters in non-linear manifolds. t-SNE and UMAP were invented to solve the "crowding problem," allowing us to visualize clustered high-dimensional data (like text embeddings or image features) on a 2D plot.

Think of It Like This

Imagine trying to unroll a crumpled up piece of paper onto a flat desk without tearing it. You want the ink marks that were touching each other on the crumpled ball to still be right next to each other on the flat paper. t-SNE and UMAP define a mathematical spring system that pulls similar points together while pushing dissimilar points apart to flatten the data.

How It Actually Works

Both algorithms rely on modeling probabilities or fuzzy topological sets:

  1. High-Dimensional Affinities (t-SNE): First, t-SNE computes the probability that two points are neighbors in the high-dimensional space using a Gaussian distribution.
  2. Low-Dimensional Affinities: It then creates a similar probability distribution in the low-dimensional space, but uses a Student's t-distribution (which has heavier tails) to prevent points from crushing together in the center.
  3. Kullback-Leibler Divergence: t-SNE minimizes the difference (KL divergence) between these two probability distributions using gradient descent.
  4. UMAP Differences: UMAP builds on similar intuition but grounds it in Riemannian geometry and algebraic topology. It uses a different cost function (Cross-Entropy) and doesn't require global normalization, making it significantly faster and better at preserving global structure alongside local clusters.

Code

import numpy as np
def high_dim_affinity(d_ij: float, sigma: float) -> float:    # Gaussian kernel for high-dimensional distance (t-SNE logic)    return np.exp(- (d_ij ** 2) / (2 * sigma ** 2))
def low_dim_affinity(d_ij: float) -> float:    # Student's t-distribution for low-dimensional distance    return 1 / (1 + d_ij ** 2)
dist = 2.0print(f"High-dim (sigma=1): {high_dim_affinity(dist, 1.0):.3f}")# -> High-dim (sigma=1): 0.135print(f"Low-dim: {low_dim_affinity(dist):.3f}")# -> Low-dim: 0.200

Watch Out For

Misinterpreting distances: In a t-SNE plot, the size of a cluster and the distance between two distinct clusters often mean nothing. The algorithm only reliably preserves local neighborhoods, so long-distance global relationships are usually distorted.

The Quick Version

  • t-SNE models high-dimensional similarities with Gaussians and low-dimensional similarities with t-distributions.
  • It optimizes the layout by minimizing KL divergence.
  • UMAP achieves similar cluster-preserving results but is mathematically grounded in topology, making it faster and better at maintaining global relationships.
  • Neither algorithm's 2D distances should be treated as exact metric distances.