SAN Side Adapter Masks

Keep CLIP frozen and run a small side net that refines masks from its side features. Eight million weights open the vocabulary.

SAN hangs a light side network beside frozen CLIP for masks

Why Does This Exist?

Tuning CLIP for masks erodes its names. SAN freezes CLIP fully and trains a side path that borrows mid features for boundaries. Mask plus text heads stay decoupled, so open-vocabulary segmentation keeps CLIP zero-shot skill.

This page covers side adaptation only. Full tuning variants are sibling pages.

The one idea you need first.

Side layers $s_l$ read frozen CLIP layers $v_l$ with cross links but no backward into $v$ . Mask head predicts $m_i$ ; text head scores $\mathrm{sim}(b_i, t_c)$ . Only $8$ M side weights move.

Think of It Like This

Scaffolding beside a finished tower

A tower stays open while scaffolding rises beside it for window work. Crews read plans through side planks but never drill the tower walls. Windows get fixed and the tower keeps its shape.

CLIP is the tower, side nets are scaffolding, and masks are windows. Where it stops: scaffolds come down after, while SAN ships with the model.

How It Actually Works

Run frozen ViT $f$ once. Run side net $s$ in parallel, fusing $v_l$ at four depths. Decode $100$ masks from $s$ features. Score with frozen text $t_c$ . Train mask BCE plus dice plus text contrast on COCO with open splits out.

In plain words: frozen eyes, light side hands, and text as names.

Worked example

Side $8$ M versus full $300$ M tuned: open mask mIoU $32.1$ versus $32.6$ on ADE $150$ , a $0.5$ gap for $3$ percent weights. Seen names hold $41.2$ mIoU because CLIP never moves. The trade favors sides for open use.

Watch Out For

Fusing too late for thin edges

Late-only fusion blurs wires and rails in semantic segmentation maps. Thin IoU sags. Fuse four depths including early high-res maps.

The Quick Version

SAN freezes CLIP and trains a side mask network.
Four fusions bring mid detail to masks.
Decoupled heads score masks by text.
8M weights nearly match full tuning open.
CLIP names stay intact for new phrases.