Skip to content
AI360XpertBeta

Promptable Segmentation

Point at any object with a click, box, or short phrase and get its exact pixel outline back. One model cuts every object with no retraining per object.

One image encoding plus a click prompt decodes to three masks, and the top predicted IoU of 0.78 picks the tight wheel mask with true IoU 0.75
One image encoding plus a click prompt decodes to three masks, and the top predicted IoU of 0.78 picks the tight wheel mask with true IoU 0.75

Why Does This Exist?

Fixed-list segmentation breaks the moment you meet a new object. A semantic segmentation model labels every pixel from a closed list it trained on. New object, new labeled set, new training run. You cannot ship that loop for every logo, tool, or tumor variant you will ever see.

Promptable segmentation flips the interface. You give a prompt (a click, a box, or a short phrase) and you get a mask (a binary image that marks the prompted object) back. No retraining per object. The heavy work happens once per photo. Each interaction stays cheap.

The trade is clear. You pay one heavy encode per image to buy millisecond responses per prompt. You lose certainty on ambiguous clicks. One point on a wheel logo can mean the logo, the wheel, or the whole car. That is why this page covers prompt-to-mask plus ambiguity handling, not fixed-class labeling. The full SAM system lives in Segment Anything. Text prompts lean on joint image-text numbers from vision-language models.

The one idea you need first. A mask is a binary image the same size as the photo: 1 where the object sits, 0 elsewhere. Overlap quality is intersection over union: area of overlap divided by area covered by either mask. Say truth covers 20 pixels, prediction covers 22, and they share 18. Union is 20+22−18=2420 + 22 - 18 = 24 and IoU is 18/24=0.7518 / 24 = 0.75. Keep that ratio in mind. It drives the whole page.

Skip this setup when your labels are closed and stable. A fixed tumor or road model with tight latency wants its own trained head, not a prompt loop.

Think of It Like This

Pointing at a print while a friend cuts

You print one sharp photo and lay it on the table. You point a finger at the wheel and say “that one.” Your friend cuts around it with scissors and hands you the cutout. Next you drag a finger box around the whole car. Same print, new cutout. Then you say “the side door.” Same print again.

Finger, box, and words map to point, box, and text prompts. The print maps to the cached image features. Scissors map to the light decoder. Three cut sizes for one vague point map to the three masks below.

Where it stops: your friend understands wheels and doors. Scissors do not. The model only cuts shapes it has learned to read from data, so a weird transparent part or a pile of overlapping objects still confuses it.

How It Actually Works

Call the photo xx. Call the heavy vision encoder ff. It runs once per photo and returns features v=f(x)v = f(x), where vv means a grid of vectors that describe shape and texture at each spot. Heavy here means slow and done one time, then cached.

Call the user input pp. It is a point (x,y)(x, y) in pixels, a box (x1,y1,x2,y2)(x_1, y_1, x_2, y_2), or a text vector from a joint embedding. Call the light prompt encoder hh. It returns a prompt vector q=h(p)q = h(p), where qq means the same request written in the decoder’s language. Light means tiny cost per click.

Call the light mask decoder gg. It fuses both streams with cross-attention and returns a mask m=g(v,q)m = g(v, q), where mm means a per-pixel yes or no for the prompted object. It also returns a score ss for each mask, where ss means predicted IoU, the model’s own guess at overlap quality without seeing truth.

Encode the image once

You run ff a single time. Every later click reuses the same vv. That split is the whole speed story. If you re-ran the big encoder per click, interaction would crawl. Cache vv and each new pp only pays hh plus gg, which is milliseconds.

Decode each prompt to three masks

One pp rarely means one thing. A click on a shirt logo can mean logo, shirt, or person. So gg outputs three masks per prompt: sub-part, part, and whole. Each carries its own ss. You pick the max ss. That ranking is the ambiguity fix. Do not trust a single mask on an ambiguous click.

Text prompts take the same path. The text side uses joint image-text vectors in the CLIP style, so “the side door” lands near door pixels in vector space. Boxes work the same way through hh. Points are the hardest case and the reason three outputs exist.

Worked example

Truth is a wheel covering 20 pixels. You click its center.

Mask A is tight around the wheel. It predicts 22 pixels and shares 18 with truth. Union is 20+22−18=2420 + 22 - 18 = 24. True IoU is 18/24=0.7518 / 24 = 0.75. Model predicts s=0.78s = 0.78.

Mask B is the whole car. It predicts 50 pixels and contains all 20 truth pixels. Union is 20+50−20=5020 + 50 - 20 = 50. True IoU is 20/50=0.4020 / 50 = 0.40. Model predicts s=0.43s = 0.43.

Mask C is the car side. It predicts 30 pixels and shares 19 with truth. Union is 20+30−19=3120 + 30 - 19 = 31. True IoU is 19/31≈0.6119 / 31 \approx 0.61. Model predicts s=0.62s = 0.62.

Ranking by ss gives 0.780.78, 0.620.62, 0.430.43. Max picks Mask A. The tight wheel wins even though one click could have meant three things. The diagram shows the same three scores on the same three shapes.

The assumption: your pp points at one visible, coherent thing. Heavy occlusion, glass, or a 6-pixel screw breaks it. Then ss stays low across all three and no pick looks clean.

Code

This snippet recomputes the three true IoUs from pixel counts and picks by predicted score, which is the exact rule the decoder uses.

def iou(pred: int, truth: int, inter: int) -> float:    union = pred + truth - inter  # count shared pixels once    return inter / union

def pick_best(scores: list[float]) -> int:    best = 0    for i in range(1, len(scores)):        if scores[i] > scores[best]:            best = i    return best

true_ious = [iou(22, 20, 18), iou(50, 20, 20), iou(30, 20, 19)]print([round(v, 2) for v in true_ious])  # -> [0.75, 0.4, 0.61]
pred_ious = [0.78, 0.43, 0.62]  # tight, whole, sidek = pick_best(pred_ious)print(k)  # -> 0print(pred_ious[k])  # -> 0.78

Index 00 with 0.780.78 selects the tight wheel. True 0.750.75 backs it up. Same numbers as the worked example and the picture.

Watch Out For

One click, wrong scale

Symptom: you click a shirt logo and get the whole person. Predicted IoU looks confident, just on the wrong level. You assume the model missed when it actually answered a different valid question.

Fix: read all three masks, not just the top one. Add a second point on the part you want, or switch to a box or short phrase. Ambiguity drops fast with one extra hint.

Tiny parts snap to the parent

Symptom: you click a small badge and the mask jumps to the door. Zoom out and the badge is a few pixels wide, below what the feature grid resolves. The click lands on neighbor features.

Fix: crop and zoom around the click, then prompt again. Judge prompt quality only at a resolution where the part spans many tokens.

The Quick Version

  • Promptable segmentation turns a click, box, or phrase into a pixel mask with no per-object training.
  • The image encodes once per photo while each prompt encodes and decodes in milliseconds.
  • One ambiguous click returns three masks for sub-part, part, and whole ranked by predicted IoU.
  • Text prompts work because image and text share joint embeddings.
  • Skip promptable models for closed vocabularies or tiny low-resolution parts without a zoomed crop.