Generative Adversarial Networks

Skip writing a loss for realism. Train a critic to spot fakes and train the maker to fool it, so the game is the loss.

Noise feeds a generator while real steel photos feed a discriminator, and the verdict gradient looping back is the only lesson the generator gets

Why Does This Exist?

A steel mill photographs sheet with a line camera to flag surface defects. It owns 40000 clean photos and 300 scratches, pits, and roll marks. Nobody will damage good steel to grow the small set.

So generate more. The obvious loss fails first. Squared error against a real defect averages all defects into a grey smudge, because no written formula says what photographic realism is. Any fixed target you can write rewards the average, not a plausible instance.

The 2014 fix stops writing the loss and learns one. A second network judges real against generated, and the first network is scored by that judge failure rate. The judge improves, the standard rises, the maker improves, the judge improves again.

Call the maker the generator: noise in, image out. Call the judge the discriminator: image in, real or fake out. This page covers that game, its sharp samples, and its three failures. The likelihood based maker with soft samples lives in variational autoencoders. The plain sampler stack it extends lives in autoencoders. The iterative refiner that later took over images lives in diffusion models.

Think of It Like This

A forger and an auctioneer who both keep getting better

A forger paints a canvas and hands it to an auctioneer who must say real or fake. Early on the auctioneer checks only the signature, so a passable signature suffices and the forger learns nothing about paint.

Then the auctioneer checks craquelure, the crack pattern of old varnish. Signatures stop helping, so the forger must learn how varnish cracks. Then pigment chemistry, then brush pressure at edges. Every closed gap forces a harder tell.

Nobody ever wrote down what a real painting is. The standard was built by two people disagreeing.

Where it stops: the forger who finds one canvas the auctioneer always passes will paint it forever. Sensible for the forger, useless for a gallery that wanted a collection. That repeat trick is mode collapse, and the story already contains it.

How It Actually Works

Call noise $z$ , generator $G$ , discriminator $D$ . $G(z)$ is a fake image. $D(x)$ is the judged chance that $x$ is real. Training is one objective pulled two ways:

\min_G \max_D \; \mathbb{E}[\log D(x)] + \mathbb{E}[\log(1 - D(G(z)))]

$\mathbb{E}$ means average over real photos and over noise draws. $D$ is trained as an ordinary classifier: real photos labelled real, generated ones fake. $G$ is trained to make that classifier wrong. That opposition is why this is a minimax game rather than a minimisation.

Neither loss measures progress. A falling generator loss can mean better fakes or a weaker judge. The two numbers describe who leads, not how good samples look, which breaks every habit built on loss curves including checkpoint choice.

Implicit density and what it buys and costs

A variational autoencoder optimises a bound on data likelihood, so it owns an explicit density you can query. A GAN never writes a probability anywhere. It owns a sampler and nothing else, which is what implicit means.

That split explains the trade practitioners feel. No averaging over plausible outputs means sharp samples where likelihood models blur. It also means no score for a held out image, no likelihood, and no principled convergence test.

Worked example: the measure that misses collapse

Take four defect types at $[-6.0, -2.0, 2.0, 6.0]$ with spread $0.4$ . Score one sample by log density under those four bumps, then average. That is realism. Count separately how many of the four bumps own at least one nearby sample. That is coverage.

Real data with eight draws near the centres scores realism about minus $0.03$ and hits 4 of 4 modes. A collapsed maker emitting only the minus $2.0$ type scores about minus $0.02$ , marginally better than real, because every sample sits dead centre of a true bump. Per sample plausibility cannot see the failure. Coverage sees it at once: 1 of 4. The diagram shows noise feeding the generator on the upper lane, real photos on the lower lane, one judge, and the verdict looping back as the only lesson.

The three failures

Mode collapse is the signature: narrow output the current judge will not reject, so quality holds while diversity dies. Your defect maker emits one convincing scratch thousands of times.

Non convergence is the chase: two networks can cycle without settling, and no loss value tells you where in the cycle you stand.

Vanishing generator gradient is the runaway judge: a discriminator far ahead rejects every fake with near total confidence, and a confident correct answer carries almost no gradient, so the maker gets nothing exactly when it needs direction. Stabilisers come in three families: objectives that keep gradient alive while the judge wins, constraints that stop the judge going too sharp, and schedules that grow resolution gradually.

Code

The point is the check: show realism missing collapse while coverage catches it, with no networks.

1import math2
3modes: list[float] = [-6.0, -2.0, 2.0, 6.0]4spread: float = 0.45
6def realism(samples: list[float]) -> float:7    total: float = 0.08    for x in samples:9        d: float = 0.010        for m in modes:11            z: float = (x - m) / spread12            d += math.exp(-0.5 * z * z)13        total += math.log(d + 1e-12)14    return total / len(samples)15
16def modes_hit(samples: list[float]) -> int:17    hit: set[int] = set()18    for x in samples:19        best: int = min(range(4), key=lambda i: abs(x - modes[i]))20        if abs(x - modes[best]) < 1.2:21            hit.add(best)22    return len(hit)23
24real: list[float] = [-6.1, -5.9, -2.1, -1.9, 2.0, 1.9, 6.0, 5.9]25collapsed: list[float] = [-2.0, -2.1, -1.9, -2.0, -2.1, -1.9, -2.0, -2.0]26print(f"{realism(real):.2f} {modes_hit(real)}")27# -> -0.03 428print(f"{realism(collapsed):.2f} {modes_hit(collapsed)}")29# -> -0.02 1

Eight real draws hit 4 of 4 with realism near minus 0.03. Eight collapsed draws sit closer to centres, so realism reads minus 0.02, better, while coverage reads 1 of 4. One number praises the failure, the pair exposes it.

Watch Out For

Mode collapse read as success because the sheet looks good

Sixty generated defects look convincing and the model ships. Two weeks later the detector catches scratches and misses roll marks, because roll marks were never generated.

Symptom: high eyeball quality with per class counts skewed against real proportions, often found only after deployment misses cluster by type. Fix: measure diversity on purpose. Cluster generated samples and count used clusters, or compare per class counts against real when labels exist, on thousands of images rather than one sheet of sixty.

Picking a checkpoint by the loss curve

Generator loss bottoms at step 40000, so that checkpoint ships, yet samples from step 25000 look better.

Symptom: the chosen checkpoint beats a weak judge cheaply instead of making better images. Fix: select outside the game with sample quality against real data plus a diversity count beside it and human review of a large sheet. Low generator loss means ahead, not good.

The Quick Version

No realism formula is written. A discriminator learns one and the generator maximises its failure.
The objective is minimax, so neither loss measures progress or picks checkpoints.
Implicit means sampler only, which buys sharp images and costs likelihoods and convergence tests.
Our four modes show eight collapsed fakes reading minus 0.02 realism, better than real minus 0.03, while covering 1 of 4.
The other failures are non convergence and vanishing gradient when the judge runs too far ahead.
Judge checkpoints on sample quality plus coverage, never on generator loss alone.