Zero-Shot Image Labels

Write each class as a sentence, embed them once, and label photos by nearest text. No training photos needed for new names.

Zero-shot classification prompts names and picks the best match

Why Does This Exist?

New classes arrive faster than labels. Zero-shot reuses joint spaces: encode names as $80$ templates like a photo of a cat, average to one text vector per class, and pick max cosine per photo. Image classification becomes naming plus matching.

This page covers prompt classification only. Open detection children add boxes.

The one idea you need first.

Text prototype $t_c$ means mean of $80$ template vectors for name $c$ . Label is $\arg\max_c \mathrm{sim}(v, t_c)$ for image $v$ . Ensembles over templates cut wording noise.

Think of It Like This

Sorting mail by reading name cards aloud

A sorter never saw tenants but holds name cards read in eighty tones each. Each letter goes to the card sounding closest. New tenants need only new cards.

Photos are letters, prompts are readings, and cosine is closeness. Where it stops: mail names stay exact, while photo names vary by phrase.

How It Actually Works

Encode photo to $v$ unit norm. Encode $80$ templates per class and mean to $t_c$ . Score cosines times scale $\gamma$ and softmax. Report top- $1$ on ImageNet and robustness sets. Tune wording per domain with a few labeled shots if allowed.

In plain words: $80$ readings per name, one match per photo, and scale $\gamma$ sharpens gaps.

Worked example

Photo scores cat $0.31$ , dog $0.26$ , fox $0.24$ . Scale $100$ : logits $31, 26, 24$ . Exponentials $2.9e13, 1.9e11, 2.6e10$ , sum $2.92e13$ . Cat probability $0.993$ . Top- $1$ over $50000$ ImageNet photos is $76.2$ percent for CLIP ViT-L, versus $75.5$ supervised ResNet-50. Matching beats one supervised net with no labels.

Watch Out For

One template deciding everything

Single a photo of a X prompts swing $5$ points by article and plural. Classes flip on wording. Always ensemble $80$ templates and test paraphrases before quoting accuracy.

The Quick Version

Zero-shot labels photos by nearest class sentence.
Eighty templates per class form the prototype.
CLIP reaches 76.2 percent top-1 on ImageNet.
Wording shifts points, so ensemble prompts.
New names need text only, no photos.