Skip to content
AI360XpertBeta

Generative Media Evaluation

A generated picture has no answer key, so you judge it like a contest entry: automatic checks for realism and prompt match, plus blind human votes. This page shows how all three scores work and where each one lies to you.

One generated image earns its verdict from three scores: FID 0.0425 for realism, CLIP 0.98 for prompt match, and an Elo upset that lifts the winner from 1500 to 1520.5
One generated image earns its verdict from three scores: FID 0.0425 for realism, CLIP 0.98 for prompt match, and an Elo upset that lifts the winner from 1500 to 1520.5

Why Does This Exist?

A classifier is easy to grade. It says "cat," the label says "cat," you tally the hits. A diffusion model hands you a picture of a cat astronaut and there is nothing to tally against. Pretty to whom? Correct according to what?

That gap bites the moment you try to ship. Version two against version one, ten thousand pictures each. Nobody can look at twenty thousand sunsets and vote honestly, yet "it looks better to me" is not a release criterion. So the field built stand-ins: automatic scores that run over thousands of images in minutes, topped up with blind human votes where taste actually matters.

This page covers the three stand-ins that survived: distribution distance for realism, embedding similarity for prompt match, and Elo-counted votes for preference. Safety and misuse are a separate lens, covered in safety evaluation. The full machinery of pairwise voting lives in pairwise preference and Elo.

The one idea you need first: an embedding turns a thing, words or pixels, into a list of numbers, and cosine similarity reads the angle between two such lists. Small angle, same direction, close meaning. You'll meet it in the CLIP section below; the full story is in vision-language models.

Think of It Like This

The county-fair pie contest

Forty pies on a long table, one blue ribbon. The organizers judge them three ways.

First, the tape measure. Someone checks crust thickness on every pie and compares the spread against last year's winners. That's FID. Notice what it never does: taste your pie. It grades the whole table, not your slice.

Second, the ticket check. Your entry card says apple. Is the pie in front of the judges apple? That's the CLIP score: did you bake what the ticket said.

Third, the blind tasting. Judges get unmarked slices, pick pairwise winners, and the results feed a leaderboard. That's Elo. Slow, hungry work, and the only part anyone trusts.

Where it stops: pies get eaten and the contest ends. Model scores stick around, and once bakers start baking to the tape measure, thickness goes perfect while flavor goes flat. Any score you optimize becomes a costume. That failure has a name and its own card below.

How It Actually Works

Three questions, three scores, in order of cost. Run the cheap automatic checks over thousands of images first. Spend human votes last, on the finalists. No single score does the job, and the rest of this section shows you computing each one by hand.

1. Is the crowd realistic? Distribution distance

Fréchet Inception Distance (FID) answers one question: do my generated images look like they came from the same world as real photos. It never scores a single picture. It compares two crowds.

The recipe: push NN real photos and NN generated images through a frozen vision encoder, so each picture becomes a feature vector. Fit one Gaussian to each cloud of vectors. Then measure the Fréchet distance between the two Gaussians:

FID=∥μr−μg∥2+Tr(Σr+Σg−2(ΣrΣg)1/2)\mathrm{FID} = \lVert \mu_r - \mu_g \rVert^2 + \mathrm{Tr}\left(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2}\right)

In plain words: μ\mu (pronounced "myoo") is the average feature vector of a crowd, the center of its cloud. Σ\Sigma (capital sigma) is the covariance, the record of how the cloud spreads in each direction. Tr\mathrm{Tr} is the trace, the sum down the diagonal. The first term punishes centers that sit apart; the second punishes spreads that differ. Lower wins, and zero means the two crowds match on average and spread.

A worked toy, diagonal spreads so every step fits on a napkin. Two feature dimensions. Real crowd: averages [0.60,0.40][0.60, 0.40], spreads [0.20,0.25][0.20, 0.25]. Generated crowd: averages [0.45,0.50][0.45, 0.50], spreads [0.30,0.25][0.30, 0.25]. With diagonal spreads the trace term collapses to squared spread gaps per dimension, so:

Dimension one: (0.60−0.45)2+(0.20−0.30)2=0.0225+0.0100=0.0325(0.60 - 0.45)^2 + (0.20 - 0.30)^2 = 0.0225 + 0.0100 = 0.0325. Dimension two: (0.40−0.50)2+(0.25−0.25)2=0.0100+0=0.0100(0.40 - 0.50)^2 + (0.25 - 0.25)^2 = 0.0100 + 0 = 0.0100. Total: 0.0325+0.0100=0.04250.0325 + 0.0100 = 0.0425.

Small, but not zero: the generator drifts left on dimension one and runs slightly wide. That's the diagram's top card.

Now the assumptions, because FID breaks loudly when they fail. It assumes Gaussian clouds and an encoder whose features match what you care about, and it assumes thousands of images per side; a few dozen make the statistics wobble. A model that memorizes its training set earns a lovely FID while inventing nothing. And scores from different encoders or sample sizes don't compare, ever. Don't use FID to crown one picture, and don't compare FIDs computed under different setups.

2. Did it draw what you asked? Prompt alignment

FID never reads your prompt. A flawless photo of a cat scores beautifully against a brief that asked for a dog. The CLIP score closes that hole: it measures the angle between the prompt's embedding tt and the image's embedding vv:

s=t⋅v∥t∥ ∥v∥s = \frac{t \cdot v}{\lVert t \rVert \, \lVert v \rVert}

Here t⋅vt \cdot v is the dot product, the coordinate-wise multiply-and-add, and ∥t∥\lVert t \rVert means the length of tt. Production embeddings arrive already unit length, so the score is just the dot product: near 11 means drawn as asked, near 00 means off brief, negative means opposite day.

Toy vectors, three dimensions, already near unit length. Prompt: t=[0.6,0.8,0.0]t = [0.6, 0.8, 0.0]. Good picture: v=[0.5,0.85,0.16]v = [0.5, 0.85, 0.16]. Dot: 0.6×0.5+0.8×0.85+0=0.30+0.68=0.980.6 \times 0.5 + 0.8 \times 0.85 + 0 = 0.30 + 0.68 = 0.98. Lengths: ∥t∥=1.0\lVert t \rVert = 1.0, ∥v∥=0.9981≈0.999\lVert v \rVert = \sqrt{0.9981} \approx 0.999. Score: 0.98/0.999≈0.9810.98 / 0.999 \approx 0.981. Wrong picture: u=[0.85,−0.5,0.17]u = [0.85, -0.5, 0.17]. Dot: 0.51−0.40=0.110.51 - 0.40 = 0.11. Length ≈1.001\approx 1.001. Score ≈0.110\approx 0.110. So 0.980.98 against 0.110.11: match versus miss, the middle card of the diagram.

Two honest footnotes. Don't compare 0.980.98 to a production dashboard: real embeddings run hundreds of dimensions wide and raw cosines land far lower. Rankings within one setup are what count. And CLIP reads like a bag of words. It notices "apple" and "table" while ignoring whether the apple is on the table, how many there are, or the word "not." Counts, positions, and negations need their own checks.

3. Would a person pick it? Blind votes and Elo

Automatic scores are cheap and gameable. The decider is people: show two pictures from two models side by side, hide the names, ask which is better, repeat. An Elo rating turns those duels into one number per model.

Each model carries a rating RR. Before a duel, the expected score for model AA against model BB is:

EA=11+10(RB−RA)/400E_A = \frac{1}{1 + 10^{(R_B - R_A)/400}}

EAE_A is the win chance the ratings imply, between 00 and 11. After the duel, with actual score SAS_A (11 for a win, 0.50.5 for a tie, 00 for a loss) and step size KK:

RA′=RA+K(SA−EA)R_A' = R_A + K(S_A - E_A)

KK sets how jumpy ratings are; take K=32K = 32, a common setting. Beat expectations and you climb; the bigger the surprise, the bigger the step.

The diagram's bottom card, worked: RA=1500R_A = 1500, RB=1600R_B = 1600, AA wins. Difference over 400400 is 0.250.25, and 100.25≈1.77810^{0.25} \approx 1.778. Expected: EA=1/(1+1.778)=1/2.778≈0.360E_A = 1 / (1 + 1.778) = 1 / 2.778 \approx 0.360. The underdog had a 36%36\% chance. Update: RA′=1500+32×(1−0.360)=1500+20.48=1520.5R_A' = 1500 + 32 \times (1 - 0.360) = 1500 + 20.48 = 1520.5. The favorite pays symmetrically: RB′=1579.5R_B' = 1579.5. Upsets move ratings most, and that's exactly the property you want when hunting for real improvements.

The price: votes cost money and patience, small samples wobble, and voters must stand in for your actual users. Arena regulars and your customers want different things. Use Elo to decide between finalists, not to score every checkpoint.

Code

Two checks you can run anywhere: cosine similarity for prompt alignment, and the Elo update for a duel result. Same numbers as the worked examples.

import math

def cosine(a: list[float], b: list[float]) -> float:    """Cosine similarity between two embedding lists."""    dot = sum(x * y for x, y in zip(a, b))    norm_a = math.sqrt(sum(x * x for x in a))    norm_b = math.sqrt(sum(y * y for y in b))    return dot / (norm_a * norm_b)

def elo_duel(r_a: float, r_b: float, score_a: float, k: float = 32.0) -> tuple[float, float]:    """New (rating_a, rating_b) after one duel. score_a is 1, 0.5, or 0."""    expected_a = 1.0 / (1.0 + 10.0 ** ((r_b - r_a) / 400.0))    shift = k * (score_a - expected_a)    return (r_a + shift, r_b - shift)

t = [0.6, 0.8, 0.0]good = [0.5, 0.85, 0.16]bad = [0.85, -0.5, 0.17]print(round(cosine(t, good), 3))  # -> 0.981print(round(cosine(t, bad), 3))  # -> 0.11
new_a, new_b = elo_duel(1500.0, 1600.0, 1.0)print(round(new_a, 1), round(new_b, 1))  # -> 1520.5 1579.5

The printout agrees with the hand computations: 0.9810.981 versus 0.110.11 for the two pictures, and 1520.51520.5 against 1579.51579.5 after the upset.

Watch Out For

Baking to the tape measure until the pictures go bland

Symptom: FID falls release after release while your own eyes report nothing. The gallery fills with safe, samey, slightly desaturated images nobody loves.

Someone on the team tuned prompts, guidance, or cherry-picking thresholds directly against FID. That's Goodhart's law wearing a lab coat: the measure became the target and stopped measuring taste. FID rewards sitting on top of the real distribution, and the dead center of any distribution is boring by definition.

Fix: freeze FID as a guardrail, never the finish line. Gate releases on "FID no worse than last time," then let blind votes crown the winner. If FID and voters disagree, believe the voters and go find out what the metric cannot see.

A glowing CLIP score on a miscounted picture

Symptom: the prompt says "three red apples on a blue table" and the winner shows two green ones, yet its CLIP score beats every faithful render. You ship it because the number said so.

CLIP matches ingredients, not arrangements. Two apples still light up "apple," green still sits near red, and the missing third apple costs almost nothing. Negations fail the same way: "no people" barely dents a crowd scene.

Fix: split alignment from aesthetics. Keep CLIP for rough relevance, then add narrow attribute checks for whatever your briefs actually constrain: a counter for counts, a VQA probe ("how many apples?") for layout, plus human spot reads on every release where wording matters. The attribute checks live outside this page on purpose; they belong to your eval harness, not to cosine similarity.

The Quick Version

  • FID measures the gap between two crowds of images, never one picture: lower means the generated set sits closer to real photos.
  • On diagonal feature statistics, FID adds squared mean gaps and squared spread gaps per dimension: the toy works out to 0.0325 plus 0.01, which is 0.0425.
  • CLIP score is the cosine between the prompt embedding and the image embedding: near 1 reads as drawn-as-asked, near 0 reads as off-brief.
  • CLIP behaves like a bag of words, so counts, positions, and negations need separate attribute checks or human spot reads.
  • Blind pairwise votes converted to Elo are the deciding signal: a 1500-rated model that upsets a 1600-rated rival gains about 20 points at K = 32.
  • Ship on all three at once. FID can be memorized, CLIP can be flattered, and voters can differ from your users.