Transformers Detect and Segment Together

Query transformers predict boxes and masks from one set of object slots, so detection and segmentation share matching and decoding.

One hundred query slots decode boxes plus masks with one matching step

Why Does This Exist?

CNN detectors need anchors plus NMS while segmenters need separate mask heads. Two pipelines double tuning and drift apart.

Queries unify them. This overview frames object detection set prediction, semantic segmentation mask decoding, and transformer detectors details.

Think of It Like This

One cast list for play plus costumes

One audition casts 100 actors. Each actor gets a role plus a costume fitting in one list, not two separate calls.

Queries are actors, boxes are roles, masks are costumes. Where it stops: miscast actors spoil both lists, just as bad queries hurt boxes and masks together.

How It Actually Works

Encoder builds image memory. Decoder updates $N = 100$ queries with self plus cross attention. Each query outputs class $c$ , box $b = (cx,cy,w,h)$ , and mask $m$ via dot with pixel features. Hungarian matching pairs predictions to truth once for all heads.

Worked example

$N = 3$ , two truths as in DETR pages. Cost matrix $[[-0.79, 1.83, 0.50],[1.83,-0.75,2.20],[0,0,0]]$ picks $(1,1),(2,2)$ with total $-1.54$ . Same assignment trains box plus mask heads, so one match serves both outputs.

Watch Out For

Tuning boxes while ignoring masks in the match cost

Box-only matching pairs small masks wrongly. Symptom: good boxes with ragged masks. Fix: include mask Dice in matching cost when masks matter.

The Quick Version

One query set decodes classes, boxes, and masks together.
Hungarian matching pairs each truth once for all heads.
Fixture total cost minus 1.54 picks correct pairs.
Shared queries cut anchors and NMS from both tasks.
Add mask cost to matching when masks lead.