Skip to content
AI360Xpert

Few-Shot and Zero-Shot Vision

Few-shot and zero-shot learning empower vision models to recognize new classes with extremely little or absolutely no training data.

Few-Shot uses a few examples, while Zero-Shot uses semantic descriptions.
Few-Shot uses a few examples, while Zero-Shot uses semantic descriptions.

Why Does This Exist?

Deep learning models typically require thousands of labeled examples per class to achieve high accuracy. However, in many real-world scenarios—such as rare disease diagnosis in medical imaging or identifying newly discovered species—collecting massive datasets is impossible. Few-shot and zero-shot learning exist to solve this data scarcity problem. They allow models to generalize to completely new, unseen classes by leveraging either a tiny "support set" of a few images or high-level semantic descriptions (like text).

Think of It Like This

Identifying Exotic Animals

If I show you exactly one picture of an okapi and tell you its name, you can probably pick an okapi out of a lineup forever after (Few-Shot). If instead I give you a written description—"it has the body of a brown horse but zebra-like stripes on its legs"—you could identify one without ever having seen a picture of it (Zero-Shot). Both techniques bypass the need to see thousands of examples.

How It Actually Works

While few-shot and zero-shot serve similar goals, their mechanisms differ:

  1. Meta-Learning (Few-Shot): Models are trained on a large variety of tasks rather than one massive dataset. They learn a general-purpose feature space where images of the same class cluster together. During inference, the model compares the query image to the 1-5 examples in the "support set" using a distance metric (like cosine similarity) and assigns it the class of the closest match.
  2. Semantic Alignment (Zero-Shot): Zero-shot learning bridges vision and language. The model is trained on image-text pairs (like CLIP) to map images and semantic descriptions into the exact same embedding space.
  3. Inference without Images (Zero-Shot): At test time, if you want to classify a new object, you embed the text descriptions of all possible classes into the shared space. You then pass the new image through the vision encoder and select the text embedding that lies closest to the image embedding.

Code

# -> Simulated Zero-Shot classification with embeddingsimport torchimport torch.nn.functional as F
def zero_shot_predict(image_embed: torch.Tensor, text_embeds: dict[str, torch.Tensor]) -> str:    best_match = None    highest_sim = -1.0        for label, text_embed in text_embeds.items():        sim = F.cosine_similarity(image_embed, text_embed, dim=0).item()        if sim > highest_sim:            highest_sim = sim            best_match = label                # -> "horse"    return best_match

Watch Out For

Domain Bias in Semantic Spaces

In zero-shot learning, the model relies entirely on the semantic descriptions it learned during pre-training. If the visual characteristics of your specific target domain differ from how the text usually describes them on the internet, the model's accuracy will plummet.

The Quick Version

  • Few-shot learning classifies new categories using only 1 to 5 labeled examples (the support set).
  • Zero-shot learning classifies new categories using zero labeled images, relying instead on text descriptions or semantic attributes.
  • Few-shot often uses metric learning or meta-learning to compare images.
  • Zero-shot aligns visual and semantic embeddings into a shared space.