Semantic Segmentation

Semantic segmentation gives every pixel a class label, producing a dense map where all pixels of one class share a label, down to exact object boundaries.

Why Does This Exist?

A bounding box around a pedestrian includes half the background sidewalk, which is fine for counting but useless for treatment planning, where the exact tumor boundary decides the dose, or for driving, where the precise road-edge line steers the car. Object detection answers where things are; surgery, autonomy, and AR overlays need the shape itself.

Semantic segmentation provides that precision: every pixel receives a class label, so sky, road, car, and pedestrian each become exact regions. Before neural nets, the same need was met with hand-built region methods covered in classical image segmentation. This page covers the dense-prediction mechanism, the encoder-decoder fix for lost resolution, and the mIoU score with a hand-scored example; full architectures get their own pages in encoder-decoder designs and the DeepLab family.

Think of It Like This

Coloring a map by district, not by house

A city map must show districts, not individual houses: every block gets exactly one district color, and two adjacent houses in one district share it. Nobody draws a box around each house first; the color goes straight onto the land, down to the boundary street.

Pixels are the blocks and classes are the districts. The analogy stops at identity: the map colors all houses in a district alike and forgets which is which, and likewise semantic segmentation labels two touching pedestrians with one shared person label instead of separating them.

How It Actually Works

From one label to a label per pixel

A classifier ends in fully connected layers that output one vector per image. Fully convolutional networks (2015) replace those layers with $1 \times 1$ convolutions, so the output becomes a $H \times W \times K$ score map: height $H$ , width $W$ , and one score per class $K$ at every pixel. The catch is resolution: backbone pooling shrinks the map 32 times over, so the network upsamples back with transposed convolutions toward full size.

Recovering boundaries with skips

Upsampling alone gives blurry edges, because the bottleneck forgot where exactly the boundary sat. U-Net (2015) fixes this with skip connections: each upsampling step concatenates the matching encoder feature map from before pooling, reuniting fine location detail with coarse class context. Dilated convolutions offer the complementary fix, spreading a $3 \times 3$ kernel over a wider area to grow context without any downsampling at all.

Scoring with mean IoU

For each class $k$ , intersection over union divides correctly predicted pixels $TP_k$ by the union of predicted and true pixels $TP_k + FP_k + FN_k$ , where $FP_k$ are false alarms and $FN_k$ are misses. Mean IoU averages that fraction over all $K$ classes, so a rare class counts exactly as much as the background.

Worked example

Take a 9-pixel toy scene with 4 true cat pixels. The model predicts 4 cat pixels, 3 of them right. Cat IoU is $3 / (4 + 4 - 3) = 3/5 = 0.6$ : three hits over five pixels anyone claimed. The 5 true background pixels meet 5 predicted background pixels with 4 agreed, so background IoU is $4 / (5 + 5 - 4) = 4/6 = 0.6667$ . Mean IoU is $(0.6 + 0.6667) / 2 = 0.6333$ . The diagram, the prose, and the snippet below all use these same nine pixels.

Code

1truth_cat, pred_cat, inter_cat = 4, 4, 32truth_bg, pred_bg, inter_bg = 5, 5, 43
4iou_cat = inter_cat / (truth_cat + pred_cat - inter_cat)5iou_bg = inter_bg / (truth_bg + pred_bg - inter_bg)6print(round(iou_cat, 4), round(iou_bg, 4), round((iou_cat + iou_bg) / 2, 4))7# -> 0.6 0.6667 0.6333

Watch Out For

Letting background pixels dominate the loss

Sky and road can own 80 percent of a driving scene while pedestrians own under 1 percent, so plain cross-entropy rewards a model that perfects sky and ignores people. The symptom is high pixel accuracy with missing small objects. Weigh classes inversely to frequency or switch to focal loss, and judge by mIoU, never by raw accuracy.

Asking semantic masks to separate touching instances

All person pixels share one label, so two overlapping pedestrians merge into a single blob. The symptom is a counting or tracking pipeline that undercounts crowds. That separation is a different task: use instance segmentation when identity matters and keep semantic masks for scene understanding.

The Quick Version

Semantic segmentation outputs an $H \times W \times K$ score map, one class per pixel.
Skip connections reunite encoder detail with decoder context for sharp boundaries.
Dilated convolutions grow context without downsampling away resolution.
Mean IoU averages per-class overlap, weighting rare classes equally with background.
Worked numbers: cat IoU $0.6$ , background IoU $0.6667$ , mIoU $0.6333$ on nine pixels.