Open Vocabulary Masks Idea

Score every mask by phrase text so new names segment with no retraining. Proposals plus language carry the tail.

Open vocabulary segmentation paints masks for any phrase

Why Does This Exist?

Fixed mask heads stop at trained names. Open pipelines propose masks class-free, then score each with text like worn armchair. ADE open splits grade unseen names. This residual page frames the task; methods live in CLIPSeg, OpenSeg, and SAN.

This idea builds on semantic segmentation plus mask transformers like Mask2Former.

The one idea you need first.

Mask $m_i$ pools features to $b_i$ ; score $\mathrm{sim}(b_i, t_c)$ picks phrases. Class-free proposals protect novel shapes. Mask AP on unseen splits is the grade.

Think of It Like This

Tracing shapes first, naming after

A child traces every shape on a page before naming any. Then a word list arrives and each tracing gets the best-fit word. New words work because tracing came first.

Masks are tracings, phrases are the list, and similarity is the match. Where it stops: paper tracings stay exact, while mask pools blur edges.

How It Actually Works

Propose $100$ masks class-free with a mask transformer. Pool region features. Score against phrase texts. Keep top masks per phrase after mask NMS. Pretrain on COCO plus caption mixes; test on ADE $150$ open splits.

In plain words: propose broadly, name by language, and grade the tail.

Worked example

Worn armchair mask IoU $0.68$ with truth, scores $0.38$ versus $0.19$ for sofa. Kept at threshold $0.25$ . Open mask AP on unseen $50$ ADE names rises from $7.2$ closed to $16.8$ open. The $9.6$ jump is language scoring at work.

Watch Out For

Proposals missing novel shapes

Base-tuned proposers drop odd shapes before scoring. Tail recall caps low. Train class-free and check proposal recall on unseen splits before tuning text heads.

The Quick Version

Open segmentation scores class-free masks by phrase text.
ADE open splits grade unseen names.
Mask transformers propose; language names.
Novel shapes need class-free training.
Phrase detail plus thresholds set precision.