CLIPSeg Text Cut Masks

Feed a phrase into a tiny decoder over frozen CLIP features and cut the matching mask. New words segment with no new heads.

CLIPSeg conditions a small decoder on text to cut one mask

Why Does This Exist?

New mask classes usually need new heads plus full retraining. CLIPSeg freezes CLIP and trains a $2$ M decoder that reads phrase vectors via FiLM modulation. Any phrase then cuts a mask, serving open-vocabulary segmentation cheaply.

This page covers text-conditioned decoding only. Proposal-scoring variants are the OpenSeg page.

The one idea you need first.

FiLM scales and shifts visual maps by $\gamma(t), \beta(t)$ from text $t$ . PhraseCut pairs $340000$ phrases to masks for training. Loss is binary mask cross entropy per pixel.

Think of It Like This

A stencil cutter reading aloud orders

A cutter hears red scarf and sets blade depth and angle from the words, then cuts that shape from patterned paper. New orders need no new machine.

CLIP paper holds the pattern, FiLM sets the blade, and the decoder cuts. Where it stops: paper patterns stay fixed, while CLIP features shift with input.

How It Actually Works

Encode photo patches $v$ with frozen CLIP and phrase $t$ with text tower. Modulate decoder layers by FiLM from $t$ . Decode to one mask $m$ at $352$ pixels. Train on PhraseCut plus COCO with dice plus BCE.

In plain words: $t$ tunes every layer, $v$ supplies detail, and $m$ is the phrase cut.

Worked example

Red scarf mask $12000$ pixels, prediction overlap $9600$ , union $14400$ . IoU $0.667$ . BCE $0.18$ plus dice $0.25$ totals $0.43$ per mask early, falling to $0.21$ as FiLM sharpens. New phrase blue tarp cuts at IoU $0.58$ with no retraining.

Watch Out For

Short vague phrases cutting blobs

Word chair alone lights every seat in semantic segmentation scenes. Masks merge instances. Use fuller phrases with color or place to split instances.

The Quick Version

CLIPSeg modulates frozen CLIP maps with phrase vectors.
A 2M decoder outputs one mask per phrase.
PhraseCut supplies phrase-mask training.
New words cut new masks directly.
Fuller phrases split instances better.