SegFormer Segments Without Positions

SegFormer pairs hierarchical transformer encoders with a light MLP decoder, so dense masks come from Glaobal context plus local overlap.

Overlapped merging plus MLP decoding gives 51 mIoU at 8M weights

Why Does This Exist?

ViT segmentation needs position codes that break on new sizes plus heavy decoders. Cityscapes tests $1024 \times 2048$ inputs where that hurts.

SegFormer drops positions and decoders. It builds on hierarchical transformers, serves semantic segmentation, and contrasts mask2former query decoding.

Think of It Like This

Sketching districts then inking with a thin pen

You sketch four map scales with overlapping pencil strokes, then ink final borders with one thin pen instead of a full paint set.

Overlap merges are pencil, Mix-FFN is local ink, and MLP is the thin pen. Where it stops: thin pens cannot fix wrong sketches, just as light decoders need good encoders.

How It Actually Works

Embed $4 \times 4$ overlap patches, merge per stage. Mix-FFN adds $3 \times 3$ depthwise inside feedforward for locality, no positions. Decoder upsamples four maps to $1/4$ , concats, and applies two MLP layers to masks.

Worked example

$512 \times 512$ gives $128 \times 128 = 16{,}384$ stage-1 tokens. Reduction $R = 8$ cuts keys to $16 \times 16 = 256$ . Pairs $16{,}384 \times 256 = 4{,}194{,}304$ versus $268$ M full, about $64$ times cheaper. Four maps upsampled to $128$ and fused give $51$ mIoU class accuracy in B2 runs at about $8$ M encoder weights.

Watch Out For

Testing larger than training without padding care

Overlap merges assume divisible sizes. Symptom: edge strip errors on odd sizes. Fix: pad to multiples of $32$ before inference and crop after.

The Quick Version

SegFormer uses overlap merges plus Mix-FFN with no position codes.
Light MLP fuses four scales to masks.
Fixture $4.19$ M pairs replace $268$ M, 64 times cheaper.
B2 reaches about 51 mIoU at 8M weights.
Pad to 32 multiples for odd test sizes.