MedSAM Promptable Medical Masks

MedSAM turns clicks and boxes into medical masks in one forward pass after training on a million image-mask pairs.

MedSAM turns one click or box into a medical mask instantly

Why Does This Exist?

Painting organs slice by slice costs hours per volume. nnU-Net automates but needs retraining per task. MedSAM (2024) fine-tunes a promptable foundation model on $1$ million medical image-mask pairs across modalities. Input is image plus point or box; output is a mask in milliseconds for annotators to accept or nudge.

Think of It Like This

A stencil cutter following fingertip taps

You tap a photo once for liver and drag a box for spleen; a cutter instantly slices stencils matching taps. Taps decide which object; the cutter decides exact edges. Bad taps still cut wrong stencils.

Prompts are taps and the decoder is the cutter. The analogy stops at 3D: slice prompts need propagation for volumes.

How It Actually Works

An image encoder runs once per scan; a prompt encoder embeds clicks, boxes, or masks; a light decoder fuses both into a mask. Training mixes CT, MRI, ultrasound, and pathology tiles. A box around liver yields Dice near $0.90$ ; one click gives near $0.82$ before a second corrective click.

Worked example

A liver slice needs masking. A box from $(100, 120)$ to $(300, 320)$ decodes in $40$ ms to Dice $0.91$ . One positive click inside plus one negative click on kidney lifts a $0.78$ first try to $0.89$ . Painting the same slice by hand takes $4$ minutes. Review still checks the falciform edge where contrast fades.

Watch Out For

Tiny polyps missed with one click

One center click on a $4$ mm polyp under-segments by $30$ percent. The symptom is shrunken small findings. Box tiny targets or add edge clicks; do not trust single clicks under $6$ mm.

The Quick Version

MedSAM makes masks from clicks and boxes without retraining.
One image encoding serves many prompts in milliseconds.
Boxes beat single clicks on tiny findings by a clear margin.
Do not accept small masks without edge clicks and review.
Annotators confirm instead of painting, which speeds datasets.