SAM Cuts Masks From Any Prompt

SAM encodes images once then decodes masks from points, boxes, or text, so one foundation model segments anything zero shot.

One image encoding plus point prompts gives three scored masks fast

Why Does This Exist?

Every new mask task trained a new model on small masks. Labeling SA-1B scale by hand would take years.

SAM splits heavy encode from light prompt. It lifts semantic segmentation zero shot, aids instance segmentation proposals, and pairs with open vocabulary boxes.

Think of It Like This

A printed photo plus tracing overlays

You print one sharp photo once, then lay thin overlays for each request: circle the dog, box the car. Printing stays once while overlays stay cheap.

Image encoder is printing, prompt decoder is overlays. Where it stops: blurry prints limit overlays, just as weak encodings cap prompt masks.

How It Actually Works

ViT-H encodes $1024 \times 1024$ to $64 \times 64 \times 256$ once. Prompt encoder embeds points, boxes, masks. Light decoder cross attends prompts to image and outputs $3$ masks plus IoU scores to handle ambiguity. Data engine grew to $11$ M photos and $1.1$ B masks.

Worked example

Point at $(0.52, 0.48)$ on a dog. Decoder outputs masks with areas $12{,}400$ , $8{,}200$ , $3{,}100$ pixels and IoU preds $0.91$ , $0.84$ , $0.62$ . Keep $0.91$ whole dog. Encode costs seconds once, decode costs $50$ ms per prompt, so $20$ prompts amortize the print.

Watch Out For

Prompting tiny structures with one point only

One point on a wire yields the pole blob instead. Symptom: coarse masks on thin parts. Fix: add negative points or a tight box for thin targets.

The Quick Version

SAM encodes once with ViT-H to $64 \times 64$ features, decodes prompts lightly.
Points, boxes, or text give three masks plus IoU to resolve ambiguity.
Fixture $0.91$ IoU picks whole dog over parts.
$11$ M photos and $1.1$ B masks power zero shot transfer.
Add negatives or boxes for thin structures.