Wasserstein GAN

The Problem

Early Generative Adversarial Networks (GANs) were notoriously difficult to train. They optimized the Jensen-Shannon (JS) divergence between the real data distribution and the generated data distribution. However, when the generated images were very poor (which is always true at the start of training), the JS divergence essentially maxed out, providing no meaningful gradient for the generator to learn from. This led to vanishing gradients and "mode collapse," where the generator would only produce a handful of identical, safe images to fool the discriminator.

The Idea

The paper proposed swapping the JS divergence for the Wasserstein distance (also known as the Earth Mover's distance). Unlike JS divergence, the Wasserstein distance provides a smooth, meaningful gradient everywhere—even when the real and generated data distributions don't overlap at all. It asks: "What is the minimum cost to physically move the 'dirt' of the generated distribution into the shape of the real distribution?"

How It Works

To use the Wasserstein distance, the architecture required several key modifications to the standard GAN setup.

Critic vs. Discriminator: In WGAN, the adversarial network doesn't output a probability (0 to 1) of whether an image is real or fake. Instead, it outputs a raw, unconstrained real number. It is renamed from a "Discriminator" to a "Critic" because it evaluates the quality of the image rather than making a binary classification.

Lipschitz Constraint: For the Wasserstein mathematics to hold true, the Critic function must be "1-Lipschitz continuous"—meaning its output cannot change too drastically when its input changes slightly.

Weight Clipping: To enforce this Lipschitz constraint, the original WGAN simply clamped the weights of the Critic network to remain within a small fixed range (e.g., [-0.01, 0.01]).

Why It Mattered

WGAN effectively cured mode collapse and the vanishing gradient problem, making GAN training significantly more stable and predictable. Crucially, the WGAN loss metric correlated with image quality: as the loss decreased, the generated images visibly improved. This gave researchers a reliable quantitative metric to track training progress, something traditional GANs severely lacked.

What Came After

While weight clipping stabilized training, it was a blunt instrument that limited model capacity. Shortly after, researchers introduced WGAN-GP (Wasserstein GAN with Gradient Penalty), which replaced weight clipping with a softer penalty on the Critic's gradients, vastly improving performance. The Wasserstein framework became the standard theoretical backing for high-quality GAN image generation for years until the eventual rise of Diffusion models.