Skip to content
AI360Xpert
Core ML
Visual explainer

Variational Autoencoders

Learn how variational autoencoders use a probabilistic latent space and the reparameterization trick to generate new data without voids.

A plain autoencoder scatters points far apart, leaving huge voids where sampling yields nothing.
A plain autoencoder scatters points far apart, leaving huge voids where sampling yields nothing.

A standard autoencoder has no reason to organize the empty space between the data points it has seen. If you pick a point at random from that void and run it through the decoder, you will get meaningless noise. The space is unconstrained.

Encoding Distributions

Instead of a fixed code, the encoder outputs a mean and a spread, describing a soft patch of probability.
Instead of a fixed code, the encoder outputs a mean and a spread, describing a soft patch of probability.

To fix the void, we change what the encoder outputs. Instead of predicting a single fixed point in the latent space, it predicts a probability distribution—specifically, a mean and a spread. This ensures that every input maps to a soft patch of valid points rather than a rigid coordinate.

Reparameterization

By sampling standard noise externally and shifting it by the mean and spread, gradients can flow back.
By sampling standard noise externally and shifting it by the mean and spread, gradients can flow back.

Sampling directly from the distribution breaks the training process because gradients cannot flow backwards through a random operation. The reparameterization trick solves this by sampling noise from outside the network, then multiplying it by the spread and adding the mean. The randomness is safely bypassed by the gradient.

The Divergence Penalty

The KL divergence term pulls the distributions together, creating a continuous space where every point maps to something real.
The KL divergence term pulls the distributions together, creating a continuous space where every point maps to something real.

A new term is added to the loss function to force all these distributions to overlap near the center. This Kullback-Leibler divergence acts as a penalty, pulling the soft patches together so that no empty voids remain. Now, any random point sampled from this region will decode into something meaningful.

Where It Breaks

In posterior collapse, the encoder plays it safe by making every input identical to the standard normal.
In posterior collapse, the encoder plays it safe by making every input identical to the standard normal.

If the divergence penalty becomes too strong, the model gives up on reconstructing the input altogether. It simply outputs the exact same standard normal distribution for every single input. This is known as posterior collapse, and it results in a model that safely minimizes its loss but learns completely nothing about the data.

The Quick Version

  • The void: Standard latent spaces are disconnected, so generation fails.
  • The fix: Encoders predict a mean and spread instead of a point.
  • The trick: Randomness is moved outside the network to allow gradients.
  • The tension: Divergence loss forces patches to overlap, filling gaps.
  • The failure: Too much divergence penalty causes posterior collapse.

What to Read Next