Skip to content
AI360Xpert
Paper Breakdowns
Paper breakdown

Segment Anything Model (SAM)

The 2023 Meta paper that introduced a promptable foundation model for image segmentation, capable of zero-shot segmentation of any object.

Paper: Segment Anything

Authors: Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, Ross Girshick · 2023

Read the paper
SAM uses a heavy image encoder to compute an embedding once, and a lightweight mask decoder that can be prompted repeatedly (with points, boxes, or text) in real-time.
SAM uses a heavy image encoder to compute an embedding once, and a lightweight mask decoder that can be prompted repeatedly (with points, boxes, or text) in real-time.

The Problem

Image segmentation (outlining the exact pixels of an object) was historically solved with bespoke models trained on specific categories. If you wanted to segment medical cells, you trained a model on cells. If you wanted to segment cars, you trained a model on cars. There was no "zero-shot" foundation model for segmentation that could take an unseen image and instantly segment a novel object without retraining.

The Idea

Meta proposed the Segment Anything Model (SAM). They reframed segmentation as a "promptable" task: the model takes an image and a prompt (a click, a bounding box, or text) and returns a valid segmentation mask. To train it, they built the SA-1B dataset, containing 11 million images and over 1 billion high-quality masks, collected via a massive data engine that combined automated generation with human-in-the-loop annotation.

How It Works

SAM's architecture is designed for real-time interaction:

  1. Image Encoder: A heavy Vision Transformer (ViT) processes the high-resolution image to produce an image embedding. This is computationally expensive but only happens once per image.
  2. Prompt Encoder: A lightweight network encodes the user's prompts (points, boxes, or text) into prompt embeddings.
  3. Mask Decoder: A very fast Transformer decoder takes the image embedding and the prompt embeddings and predicts the final segmentation mask. Because the heavy lifting was done by the Image Encoder, the Mask Decoder can run in a few milliseconds in a web browser, enabling real-time interactive prompting.

Why It Mattered

SAM became the "GPT-3 moment" for computer vision segmentation. It generalized so well to unseen domains (like microscopic imagery or underwater photos) that it effectively solved interactive segmentation as a general task, replacing thousands of custom, domain-specific models.

What Came After

SAM became a fundamental building block in larger vision pipelines. It was used to generate training data for other models, track objects in video (SAM 2), and act as the visual grounding layer for large multimodal models. Its "promptable" interface became the standard for interacting with vision foundation models.