GAN Evaluation

GAN scores compare the statistics of generated and real image sets, so FID near zero means matching distributions while high scores expose the gap.

FID compares real and generated feature clouds by their means and spreads, shrinking to zero only when both match

Why Does This Exist?

Eyeballing samples does not scale and does not replicate. Two researchers looking at the same fifty faces disagree about which model wins, and cherry-picked grids flatter every method. Training needs a number to minimise and the field needs a number to rank by.

Feature-statistics metrics fill the gap. Instead of comparing pixels, they pass thousands of real and generated images through a fixed classifier and compare the resulting feature clouds. Distance between clouds measures both quality and coverage at once. The failure mode they catch most famously is mode collapse, where sharp samples hide missing variety.

Think of It Like This

Comparing two orchards by their harvest baskets

Two orchards claim equal harvests. Rather than tasting one apple each, you collect a thousand apples per orchard and compare the baskets: average weight, spread of sizes, mix of varieties. Matching baskets means matching harvests even if no single pair of apples is identical.

Inception features are the weigh-and-sort station. FID compares the baskets' statistics, not individual apples. The check fails when the station itself is biased, sorting only by redness while one orchard grows green apples, which is the feature-bias limit below.

How It Actually Works

Pass real images $x$ and generated images $g$ through Inception-v3 up to the pool layer, giving feature vectors. Fit a Gaussian to each cloud with means $\mu_r, \mu_g$ and covariances $\Sigma_r, \Sigma_g$ . The Frechet Inception Distance is:

\mathrm{FID} = \lVert \mu_r - \mu_g \rVert^2 + \mathrm{Tr}\big(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2}\big)

The first term punishes shifted averages, the trace term punishes mismatched spreads. Inception Score (IS) instead scores generated images alone: it is high when each image classifies confidently and the set covers all classes evenly. Kernel Inception Distance (KID) replaces the Gaussian assumption with a polynomial-kernel MMD, which stays unbiased on small sample sets where FID wobbles.

Worked example

Two-dimensional features, both clouds with identity covariance, real mean $(0, 0)$ , generated mean $(0.5, 0)$ . Mean shift squared is $0.25$ ; the trace term is $\mathrm{Tr}(I + I - 2I) = 0$ ; FID $= 0.25$ . For IS with two classes where every image scores $0.9$ for class A: $\mathrm{KL} = 0.9 \log(0.9/0.5) + 0.1 \log(0.1/0.5) = 0.529 - 0.161 = 0.368$ , so IS $= e^{0.368} \approx 1.45$ , far from the ideal $2.0$ because class B never appears.

Code

FID on toy clouds, showing both terms separately.

1import math2
3
4def fid_toy(mu_r: list[float], mu_g: list[float]) -> float:5    """FID for identity covariances: mean shift squared, trace term zero."""6    return sum((a - b) ** 2 for a, b in zip(mu_r, mu_g))7
8
9print(fid_toy([0.0, 0.0], [0.5, 0.0]))  # -> 0.2510print(round(math.exp(0.9 * math.log(1.8) + 0.1 * math.log(0.2)), 4))  # -> 1.4453 toy IS

Watch Out For

Comparing FIDs computed with different feature layers

FID has no units and no absolute scale; its value depends on the exact network layer, image resizing, and sample count. A score of $12$ from one pipeline and $9$ from another may rank models oppositely to a matched comparison.

Fix one pipeline for every comparison: same checkpoint, same resize, same sample count, ideally $50{,}000$ per side. Report the pipeline alongside the number or the number means nothing.

High IS from a collapsed but confident model

A generator emitting one perfect poodle per class scores well on confidence while covering nothing within each class. IS rewards the sharpness and the class balance but never looks inside a class.

Pair IS with FID or recall metrics. A high IS beside a poor FID is the signature of confident, narrow generation.

The Quick Version

FID fits Gaussians to real and generated Inception features and measures their Frechet distance.
Our toy shift gives FID $0.25$ , all from the mean term with zero trace contribution.
Inception Score rewards confident, evenly spread classifications; our toy gives $1.45$ of $2.0$ .
KID uses kernel MMD instead, staying reliable on small samples.
Never compare scores across pipelines, and never trust IS without a coverage metric.