Latent Spaces and Manifold Learning

Why Does This Exist?

Here is a claim worth taking literally, because most of generative modelling follows from it. Real data occupies a vanishingly small, curved, low-dimensional subset of the space it is stored in.

Count it. A 100 by 100 greyscale image is 10,000 numbers, each 0 to 255, so there are $256^{10000}$ possible images, a 1 followed by about 24,082 zeros. Sample one uniformly at random and you get static. Sample every microsecond for the age of the universe and you still get static, every time. You will never hit a face, a cat, or a page of text.

So the set of images that look like anything is unimaginably tiny inside the space that holds it. And it is smooth: rotate a head two degrees and you are still in the set of faces, brighten it slightly and you are still in it. A tiny, smooth, curved subset is what mathematicians call a manifold, and the claim above is the manifold hypothesis. The intrinsic dimension is how many numbers actually vary. A photo of one person under studio lighting might be described by pose, expression, lighting angle and a handful of other things, perhaps 30 numbers, stored as 10,000.

This page is about the coordinate system on that sliver: what a latent space is, why distance there beats distance in storage, and why generation lives there. The linear projection method that finds flat slivers is dimensionality reduction. The lookup vectors you actually store and search are vector embeddings. The sampling machinery that uses the space to make new rows is synthetic data generation.

The one idea you need first. A vector is a list of numbers, and distance between two vectors is a single number saying how far apart they sit. Pixel distance counts per-position disagreements. Latent distance counts disagreements along the surface. That difference is the whole page, and the practical face of it is vector embeddings.

Think of It Like This

How It Actually Works

A model that compresses data learns three objects. An encoder $f$ maps an input $x$ to a short vector $z = f(x)$, the latent code. A decoder $g$ maps a code back to data, $\hat{x} = g(z)$. The set of codes the decoder handles well is the latent space, with intrinsic dimension $d$, the small count of numbers that actually vary. The bottleneck has to be narrow: forcing 10,000 numbers through 30 makes the model find the 30 that matter. A latent space is what an embedding is a point in.

Why latent distance beats input distance

Take a photo and shift every pixel one column to the right. In pixel space that is an enormous distance, nearly every pixel changed. To you it is the same photograph, and in a good latent space the two codes sit almost on top of each other.

That single fact is the argument for the whole approach. Distance in the stored representation measures storage accidents. Distance on the manifold measures the thing you care about.

Interpolation is where generation lives

Average two face images pixel by pixel and you get a double exposure, a ghost, not a face, because the midpoint of two points on a curved surface is not on the surface. Average their latent codes and decode, and you get a plausible third face.

That is why the latent space is the working surface for generation. Diffusion models and VAEs both sample there rather than in pixel space, for exactly this reason.

Disentanglement is the property where one latent axis moves one human-meaningful attribute. It is genuinely useful and it mostly does not happen by default. The axes a model picks are whatever made compression easiest, and they arrive tangled.

Off the manifold, a model is guessing

A model is fitted where data lives. Hand it a point off the manifold and it produces something, confidently, because nothing in the architecture reports it has never seen this region.

Two consequences matter in practice. Adversarial examples are off-manifold points a hair away from real ones, which is why a few imperceptibly changed pixels can flip a label. And interpolating in feature space invents rows that may sit in an impossible region: this is exactly the caveat on SMOTE, which builds a synthetic minority row partway between two real ones and can land it inside the majority class. Mixup gets away with the same operation because it interpolates the label too, so the model is told the point is a blend rather than a member.

Useful flip side: reconstruction error is an out-of-distribution detector. Encode, decode, measure the difference. Points on the manifold come back cleanly. Points off it do not.

Worked example

Take the unit circle, a one-dimensional manifold sitting in a two-dimensional plane. Every point is exactly one number, its angle $\theta$, with coordinates $(\cos\theta, \sin\theta)$ and radius exactly 1.

Two real points at 0 and 90 degrees: $(1, 0)$ and $(0, 1)$. Average their coordinates and the midpoint is $(0.5, 0.5)$, whose radius is $\sqrt{0.5^2 + 0.5^2} = \sqrt{0.5} \approx 0.7071$. It sits about 29 percent inside the circle, which is to say it is not on the manifold and is not data.

Average their angles instead: 45 degrees, coordinates $(\cos 45, \sin 45) \approx (0.7071, 0.7071)$, radius exactly 1.0. Still on the circle.

Same two points, same halfway instruction. One route stays in the data and one leaves it, and the only difference is which coordinate system you did the arithmetic in.

Code

The point of this snippet is the check: the chord midpoint fails the radius test while the angle midpoint passes it.

1import numpy as np2
3
4def radius(p: np.ndarray) -> float:5    """Distance from the centre. On the manifold this is exactly 1."""6    return float(np.linalg.norm(p))7
8theta: np.ndarray = np.array([0.0, np.pi / 2])9pts: np.ndarray = np.stack([np.cos(theta), np.sin(theta)], axis=1)10chord: np.ndarray = pts.mean(axis=0)11arc: np.ndarray = np.array([np.cos(np.pi / 4), np.sin(np.pi / 4)])12print(np.round(pts, 3).tolist())      # -> [[1.0, 0.0], [0.0, 1.0]]13print(np.round(chord, 4).tolist())    # -> [0.5, 0.5]14print(round(radius(chord), 4))        # -> 0.707115print(round(radius(arc), 4))          # -> 1.016print(int(round(10_000 * np.log10(256))))  # -> 24082

chord is a real pair of numbers that is not a member of the set. That is the whole failure mode, in two dimensions where you can see it.

Watch Out For

The Quick Version

• Real data occupies a tiny, smooth, curved subset of the space that stores it. Sample a 100 by 100 image uniformly and you get static, every time, out of about $10^{24082}$ possibilities.
• That subset is a manifold, and the intrinsic dimension is how many numbers actually vary, perhaps 30 of the 10,000 you store.
• A latent space is the coordinate system a model learns on that surface. The encoder maps onto it, the decoder maps back.
• A one-pixel shift is enormous in pixel space and tiny in latent space, which is the whole argument for working there.
• Interpolating latent codes gives plausible new examples. Interpolating inputs gives a double exposure, because the chord leaves the surface.
• On the unit circle, the coordinate midpoint of the 0 and 90 degree points has radius 0.7071, about 29 percent inside. The angle midpoint sits exactly on it.
• Off-manifold inputs get confident meaningless answers, which is what adversarial examples exploit and what SMOTE risks.
• Reconstruction error is a usable out-of-distribution detector.
• Latent axes are not named attributes, and coordinates from two training runs are not comparable.

What to Read Next

• Dimensionality Reduction
• Vector Embeddings
• Synthetic Data Generation