Captioning Scores CIDEr BLEU

N-gram overlap scores with consensus weighting that price generated captions against human references.

Caption scores compare generated words to reference sets, so rare shared phrases count more than filler

Why Does This Exist?

Captions have many right answers, so exact-match accuracy fails. Scoring must reward shared meaning across references while downweighting filler. BLEU prices n-gram precision with brevity penalty, METEOR adds stemming and synonyms with alignment, CIDEr weights rare n-grams by TF-IDF consensus, so captioning can compare on shared references.

This page shows toy overlap math and gaming traps. Say less about exact constants and use the official scorers for reported numbers.

Think of It Like This

Grading recipes by shared phrases

Five cooks describe one cake. Shared rare phrases like burnt sugar crust count more than a, the, and.

That rarity bonus is CIDEr. BLEU counts matching chunks, METEOR forgives baked versus baking.

How It Actually Works

BLEU averages n-gram precisions with a short-output penalty, higher is better. METEOR aligns unigrams with synonym help then penalizes fragmentation, higher is better. CIDEr cosine-scores TF-IDF n-gram vectors against references, higher is better. Report all three because each games differently.

Toy overlap walkthrough

Take reference words {dog, runs, park} and candidate {dog, runs, fast}. Unigram precision is $2 / 3 \approx 0.667$ , recall $2 / 3 \approx 0.667$ . Filler-free overlap drives the score, which is why rare shared content words decide CIDEr.

Code

Unigram precision and recall, the unit behind overlap scores.

1def unigram_pr(pred: list[str], ref: list[str]) -> tuple[float, float]:2    hits = len(set(pred) & set(ref))3    return round(hits / max(len(pred), 1), 3), round(hits / max(len(ref), 1), 3)4
5
6print(unigram_pr(["dog", "runs", "fast"], ["dog", "runs", "park"]))7# -> (0.667, 0.667)

Two shared of three give 0.667 both ways.

Watch Out For

Gaming n-grams while meaning drifts

Repeating reference phrases lifts BLEU without describing the image. The symptom is a high score with wrong scenes. Fix it with human checks and CIDEr plus SPICE-style relation views beside BLEU.

The Quick Version

BLEU prices n-gram precision with brevity penalty, higher is better.
METEOR aligns with synonyms then penalizes fragmentation.
CIDEr weights rare n-grams by consensus across references.
Toy 2 shared of 3 gives about 0.667 precision and recall.
Use official scorers and pair numbers with human looks.