Promptable Segmentation
Point at any object with a click, box, or short phrase and get its exact pixel outline back. One model cuts every object with no retraining per object.
Why Does This Exist?
Fixed-list segmentation breaks the moment you meet a new object. A semantic segmentation model labels every pixel from a closed list it trained on. New object, new labeled set, new training run. You cannot ship that loop for every logo, tool, or tumor variant you will ever see.
Promptable segmentation flips the interface. You give a prompt (a click, a box, or a short phrase) and you get a mask (a binary image that marks the prompted object) back. No retraining per object. The heavy work happens once per photo. Each interaction stays cheap.
The trade is clear. You pay one heavy encode per image to buy millisecond responses per prompt. You lose certainty on ambiguous clicks. One point on a wheel logo can mean the logo, the wheel, or the whole car. That is why this page covers prompt-to-mask plus ambiguity handling, not fixed-class labeling. The full SAM system lives in Segment Anything. Text prompts lean on joint image-text numbers from vision-language models.
The one idea you need first. A mask is a binary image the same size as the photo: 1 where the object sits, 0 elsewhere. Overlap quality is intersection over union: area of overlap divided by area covered by either mask. Say truth covers 20 pixels, prediction covers 22, and they share 18. Union is and IoU is . Keep that ratio in mind. It drives the whole page.
Skip this setup when your labels are closed and stable. A fixed tumor or road model with tight latency wants its own trained head, not a prompt loop.
Think of It Like This
Pointing at a print while a friend cuts
You print one sharp photo and lay it on the table. You point a finger at the wheel and say “that one.” Your friend cuts around it with scissors and hands you the cutout. Next you drag a finger box around the whole car. Same print, new cutout. Then you say “the side door.” Same print again.
Finger, box, and words map to point, box, and text prompts. The print maps to the cached image features. Scissors map to the light decoder. Three cut sizes for one vague point map to the three masks below.
Where it stops: your friend understands wheels and doors. Scissors do not. The model only cuts shapes it has learned to read from data, so a weird transparent part or a pile of overlapping objects still confuses it.
How It Actually Works
Call the photo . Call the heavy vision encoder . It runs once per photo and returns features , where means a grid of vectors that describe shape and texture at each spot. Heavy here means slow and done one time, then cached.
Call the user input . It is a point in pixels, a box , or a text vector from a joint embedding. Call the light prompt encoder . It returns a prompt vector , where means the same request written in the decoder’s language. Light means tiny cost per click.
Call the light mask decoder . It fuses both streams with cross-attention and returns a mask , where means a per-pixel yes or no for the prompted object. It also returns a score for each mask, where means predicted IoU, the model’s own guess at overlap quality without seeing truth.
Encode the image once
You run a single time. Every later click reuses the same . That split is the whole speed story. If you re-ran the big encoder per click, interaction would crawl. Cache and each new only pays plus , which is milliseconds.
Decode each prompt to three masks
One rarely means one thing. A click on a shirt logo can mean logo, shirt, or person. So outputs three masks per prompt: sub-part, part, and whole. Each carries its own . You pick the max . That ranking is the ambiguity fix. Do not trust a single mask on an ambiguous click.
Text prompts take the same path. The text side uses joint image-text vectors in the CLIP style, so “the side door” lands near door pixels in vector space. Boxes work the same way through . Points are the hardest case and the reason three outputs exist.
Worked example
Truth is a wheel covering 20 pixels. You click its center.
Mask A is tight around the wheel. It predicts 22 pixels and shares 18 with truth. Union is . True IoU is . Model predicts .
Mask B is the whole car. It predicts 50 pixels and contains all 20 truth pixels. Union is . True IoU is . Model predicts .
Mask C is the car side. It predicts 30 pixels and shares 19 with truth. Union is . True IoU is . Model predicts .
Ranking by gives , , . Max picks Mask A. The tight wheel wins even though one click could have meant three things. The diagram shows the same three scores on the same three shapes.
The assumption: your points at one visible, coherent thing. Heavy occlusion, glass, or a 6-pixel screw breaks it. Then stays low across all three and no pick looks clean.
Code
This snippet recomputes the three true IoUs from pixel counts and picks by predicted score, which is the exact rule the decoder uses.
def iou(pred: int, truth: int, inter: int) -> float: union = pred + truth - inter # count shared pixels once return inter / union
def pick_best(scores: list[float]) -> int: best = 0 for i in range(1, len(scores)): if scores[i] > scores[best]: best = i return best
true_ious = [iou(22, 20, 18), iou(50, 20, 20), iou(30, 20, 19)]print([round(v, 2) for v in true_ious]) # -> [0.75, 0.4, 0.61]
pred_ious = [0.78, 0.43, 0.62] # tight, whole, sidek = pick_best(pred_ious)print(k) # -> 0print(pred_ious[k]) # -> 0.78Index with selects the tight wheel. True backs it up. Same numbers as the worked example and the picture.
Watch Out For
One click, wrong scale
Symptom: you click a shirt logo and get the whole person. Predicted IoU looks confident, just on the wrong level. You assume the model missed when it actually answered a different valid question.
Fix: read all three masks, not just the top one. Add a second point on the part you want, or switch to a box or short phrase. Ambiguity drops fast with one extra hint.
Tiny parts snap to the parent
Symptom: you click a small badge and the mask jumps to the door. Zoom out and the badge is a few pixels wide, below what the feature grid resolves. The click lands on neighbor features.
Fix: crop and zoom around the click, then prompt again. Judge prompt quality only at a resolution where the part spans many tokens.
The Quick Version
- Promptable segmentation turns a click, box, or phrase into a pixel mask with no per-object training.
- The image encodes once per photo while each prompt encodes and decodes in milliseconds.
- One ambiguous click returns three masks for sub-part, part, and whole ranked by predicted IoU.
- Text prompts work because image and text share joint embeddings.
- Skip promptable models for closed vocabularies or tiny low-resolution parts without a zoomed crop.