Image Captioning

Image captioning models act as a bridge between vision and language, taking an image as input and generating a coherent textual description of its contents.

An encoder turns the photo into grid features, cross-attention points at dog pixels for the word dog, and the decoder writes left to right until the end token

Why Does This Exist?

While image classification can say a picture contains a dog and a disc, it cannot say a golden retriever catches a disc mid-air. Real uses need objects plus attributes plus relations in words: screen readers, search indexes, and chat agents that start from pixels and end in sentences. Captioning crosses that gap.

This page covers encoder-decoder captioning with attention. Joint scoring of pairs lives in image-text retrieval, object lists in object detection, and joint pretraining in multimodal learning.

Think of It Like This

The reporter and the typist

A visual analyst studies a photo and lists facts: dog, jumping, disc, park. A writer who never saw the photo takes that list and writes a smooth sentence using grammar alone. Captioning joins the two experts in one pipeline.

The encoder is the analyst, the decoder is the writer, and attention is the writer pointing at list lines per word. Where it stops: human writers ask follow-ups and revise, while decoders commit left to right with no revision pass.

How It Actually Works

An encoder $f$ turns pixels $x$ into a feature grid $v$ with a CNN or ViT backbone. A transformer decoder $g$ writes words $y_1$ through $y_T$ one by one: at step $t$ it attends over $v$ plus the past words $y_1$ to $y_{t-1}$ and predicts $y_t$ with cross-attention, so the word dog looks at dog pixels. Training minimizes cross-entropy over the $T$ positions, and generation stops at the end token. Beam search can replace greedy picks at test time.

In plain words: $v$ holds the scene, $y_{<t}$ holds the sentence so far, and attention picks the spot per word.

Worked example

Suppose the next word after a golden retriever catching a scores dog $0.62$ , cat $0.21$ , disc $0.09$ in unnormalized terms with exponentials $1.86$ , $1.23$ , $1.09$ over sum $4.18$ . Normalized shares are $1.86 / 4.18 \approx 0.445$ , $1.23 / 4.18 \approx 0.294$ , $1.09 / 4.18 \approx 0.261$ . Pick dog with loss $-\log(0.445) \approx 0.810$ . Attention weights $[0.70, 0.20, 0.10]$ over dog, grass, and sky patches sum to $1.0$ and point at the dog for this word.

Code

Greedy decoding from per-step scores with only the standard library:

1def greedy_caption(scores: list[list[float]], vocab: list[str]) -> str:2    words: list[str] = []  # picked ids so far3    for row in scores:  # one score row per step4        best = max(range(len(row)), key=lambda i: row[i])5        if vocab[best] == "<EOS>":  # stop token ends it6            break7        words.append(vocab[best])8    return " ".join(words)9
10
11vocab = ["a", "dog", "catches", "disc", "<EOS>"]12scores = [[2.0, 0.2, 0.1, 0.1, 0.0], [0.1, 2.1, 0.2, 0.1, 0.0], [0.1, 0.2, 1.9, 0.3, 0.0]]13print(greedy_caption(scores, vocab))  # -> a dog catches

Output a dog catches shows left-to-right picks with no search and no revision.

Watch Out For

Hallucinating from strong language priors

Symptom: fluent captions mention objects that are not in the photo, because the decoder repeats common text habits such as kitchen gear instead of reading pixels. Fix: check attention maps per noun and add grounding tests before trusting nouns in accessibility uses.

Training on truth but serving on guesses

Symptom: fine sentence starts with garbled endings, because training feeds true past words while serving feeds the model's own past words, so one early slip cascades. Fix: sample the model's own predictions during training and decode with beam search at test time.

The Quick Version

Image captioning maps photos to sentences with an encoder, a decoder, and cross-attention.
Words generate left to right until an end token, with no revision pass.
Attention links each word to image spots, such as $0.70$ weight on the dog patch.
The fixture picks dog at $0.445$ with loss $0.810$ for the worked step.
Strong language priors hallucinate objects, and truth-fed training splits from guess-fed serving.