WGAN and WGAN-GP

WGAN swaps the GAN coin flip for a critic score gap, and the gradient penalty keeps that critic honest so training stops collapsing.

The critic scores real photos 0.8 and fakes 0.2 for a 0.6 gap, and the penalty on the interpolated point adds 0.9

Why Does This Exist?

Classic GANs grade with a coin flip: real or fake, one sigmoid, done. When the discriminator gets good early, that sigmoid saturates and the generator gets gradients near zero. Training stalls while samples still look bad, and nobody can tell from the loss whether the model improves. The number people watch stops meaning anything.

WGAN replaces the flip with a gap. Its critic outputs raw scores with no sigmoid, and the training signal is mean real score minus mean fake score. That gap tracks sample quality through the whole run. DCGAN gives the convolutional body this game runs in, mode collapse is the failure the gap exposes early, and GAN evaluation is where you confirm the improvement with real metrics.

Think of It Like This

Two piles of soil and a moving quote

A contractor quotes moving soil from one pile to shape another. The price is mass times distance: nearby piles cost little, far piles cost a lot. That quote is the Wasserstein distance. A cheap quote means the fake pile already sits near the real one.

Weight clipping was a rule that the quote per mile must stay small, enforced by capping the trucks. The gradient penalty instead fines any quote that changes too fast along the road between piles. Same honesty, but the trucks keep full power. The analogy stops at the math: soil movers do not backpropagate.

How It Actually Works

The critic $f$ assigns a raw score to each image, no sigmoid at the end. The generator tries to raise fake scores while the critic tries to widen the gap:

\text{gap} = E[f(\text{real})] - E[f(\text{fake})]

In plain words: average critic score on real photos minus average score on generated ones. The critic maximises it, the generator maximises fake scores to close it. For this price to equal a true distance, $f$ must stay 1-Lipschitz, meaning its slope never exceeds 1 anywhere. Optimal transport is the full treatment of that distance idea.

WGAN enforced the slope cap by clipping every weight into $[-0.01, 0.01]$ . It works but cripples capacity: weights pile at the bounds and gradients thin out. WGAN-GP drops clipping and adds a soft fine. Sample an interpolated point $\hat{x}$ on the line between a real and a fake image, measure the gradient norm there, and penalise drift from 1:

\text{penalty} = \lambda \cdot (\lVert \nabla f(\hat{x}) \rVert - 1)^2, \qquad \lambda = 10

Worked example

Critic means: real $0.8$ , fake $0.2$ . Gap $= 0.8 - 0.2 = 0.6$ . At one interpolated point the gradient norm is $1.3$ . Penalty $= 10 \times (1.3 - 1)^2 = 10 \times 0.09 = 0.9$ . Critic loss $= 0.2 - 0.8 + 0.9 = 0.3$ . You can read training directly: gap rising toward larger values means fakes approaching reals, while the $0.9$ tells you the slope strays and the fine bites.

Code

The two numbers that tell you if a WGAN-GP step behaves, computed from batch means and one gradient norm.

1def wgan_gp_step(real_mean: float, fake_mean: float, grad_norm: float, lam: float = 10.0) -> tuple[float, float, float]:2    """Gap, penalty, and critic loss from batch statistics."""3    gap = real_mean - fake_mean  # higher means fakes closer to reals4    penalty = lam * (grad_norm - 1.0) ** 2  # soft 1-Lipschitz fine5    loss = fake_mean - real_mean + penalty  # critic minimises this6    return gap, penalty, loss7
8
9print(wgan_gp_step(0.8, 0.2, 1.3))  # -> (0.6, 0.9, 0.3)

Watch Out For

Keeping batch normalisation in the critic

Batch normalisation ties each interpolated point to its batch mates, so the gradient at $\hat{x}$ no longer measures the critic alone. The penalty then fines batch statistics instead of slope, and training wobbles exactly where GP should steady it.

Use layer normalisation or no normalisation in the critic. The symptom is a penalty that refuses to settle near zero while samples stay muddy.

Tuning lambda before the loss makes sense

Teams often grid-search $\lambda$ while the gap itself falls, which is backwards. If the gap collapses toward zero, the critic is too weak or the learning-rate balance is off, and no penalty weight fixes that.

First get the gap rising with $\lambda = 10$ . Only then adjust. A healthy run shows gap climbing and penalty small beside it.

The Quick Version

WGAN replaces the saturating real-or-fake loss with a critic score gap that tracks quality.
The critic must stay 1-Lipschitz, meaning its slope never exceeds 1.
Weight clipping enforces the cap crudely and thins capacity.
WGAN-GP fines $(\lVert \nabla f \rVert - 1)^2$ with weight $10$ on interpolated points instead.
Worked numbers: gap $0.6$ , penalty $0.9$ , critic loss $0.3$ .
Rising gap means progress; keep batch normalisation out of the critic.