OpenSeg Word Pixel Masks

Learn word-to-pixel matches from captions, then group pixels into masks that answer text queries. Captions teach masks without dense labels.

OpenSeg aligns words to pixels then groups masks by text

Why Does This Exist?

Dense mask labels cover few names. Captions cover many words with mouse traces. OpenSeg pretrains word-pixel scores on Localized Narratives, then tunes mask grouping on COCO. Open-vocabulary segmentation and panoptic segmentation gain rare names cheaply.

This page covers caption-to-mask grouping only. Phrase decoders are the CLIPSeg page.

The one idea you need first.

Word-pixel loss matches each noun to traced pixels. Mask loss later groups pixels into $100$ proposals scored by text. Nouns become the class list at test time.

Think of It Like This

Learning counties from travel stories with maps

Travel stories name towns with traced routes on a map. Readers first link words to map spots, then draw county borders that group spots by name. New town names map without resurveying.

Captions are stories, traces are links, and masks are counties. Where it stops: maps stay fixed scale, while photos vary in pose and size.

How It Actually Works

Encode words $w$ and pixels $v$ . Pay grounding loss on narrative traces. Add mask transformer proposing $m_i$ ; pool to $b_i$ and score $\mathrm{sim}(b_i, t_c)$ . Tune on COCO masks with open splits held out.

In plain words: captions teach words to pixels, masks group pixels, and text scores groups.

Worked example

Narrative word canoe links to $800$ traced pixels of $200000$ . Pixel accuracy $0.74$ early. After grouping, canoe mask IoU $0.61$ on held-out names. Open panoptic quality rises $8$ points over closed baselines in standard runs.

Watch Out For

Trace noise blurring small masks

Mouse traces wobble and small items blur across neighbors. Fine masks in semantic segmentation tests smear. Filter traces by dwell time and upsample features before grouping.

The Quick Version

OpenSeg pretrains word-pixel links on captioned traces.
Mask proposals group pixels for text scoring.
COCO tunes grouping with open names held out.
Captions supply rare names cheaply.
Trace filtering protects small masks.