OneFormer Unifies Three Segmentations

OneFormer conditions queries on a task token, so one model does semantic, instance, and panoptic without retraining.

Task tokens switch 200 queries across three segmentations in one model

Why Does This Exist?

Three segmentation tasks train three models on overlapping data. Serving triples cost and fixes diverge.

OneFormer shares all but the task. It unifies semantic segmentation, instance segmentation, and panoptic segmentation via mask2former decoding.

Think of It Like This

One printer with three paper trays

One printer holds letter, photo, and label trays. A tray switch changes output while the engine stays shared.

Task tokens are tray switches, queries are the engine. Where it stops: wrong tray wastes prints, just as wrong task token spoils masks.

How It Actually Works

Learn three task tokens $T_{sem}, T_{ins}, T_{pan}$ . Prepend the active one to $200$ object queries. Decode with masked attention over pyramid features. Sample task uniformly per batch with task specific matching costs.

Worked example

Batch picks panoptic. Queries $200$ plus $T_{pan}$ decode $100$ thing plus $53$ stuff masks on COCO. Loss $2.0$ class plus $5.0$ mask Dice guides both. Swap token to $T_{sem}$ at test and the same $200$ queries emit $53$ stuff maps with no extra weights.

Watch Out For

Forgetting task token at inference time

Defaulting to one task for all prompts flattens instance splits. Symptom: panoptic quality drops while semantic holds. Fix: pass the task string explicitly and validate all three heads.

The Quick Version

OneFormer adds a task token to switch semantic, instance, and panoptic.
Two hundred shared queries decode all three with masked attention.
Joint sampling trains one model instead of three.
Fixture same queries emit 153 panoptic or 53 semantic maps.
Pass task explicitly or heads mismatch.