Semantic Segmentation

Semantic segmentation assigns a class label to every single pixel in an image, producing a dense map where pixels belonging to the same object class share the same label — going beyond bounding boxes to pixel-precise understanding.

Why Does This Exist?

Bounding boxes are a crude approximation of object shape. A bounding box around a person walking includes half the background sidewalk. For medical image analysis (where the exact tumor boundary determines treatment), autonomous driving (where the precise road-sidewalk boundary matters), and AR overlays (which must match object contours), pixel-level labels are necessary.

Semantic segmentation provides that precision: every pixel receives a class label, enabling applications that require exact shape understanding.

Think of It Like This

Coloring a line drawing by region

A coloring book page has outlines of objects. Semantic segmentation is like filling in every region with a specific color: all sky pixels get blue, all grass pixels get green, all person pixels get red, all road pixels get grey. Unlike object detection, you're not drawing new bounding boxes — you're classifying every pixel that already exists in the image, down to the boundary.

How It Actually Works

Fully Convolutional Networks (FCN)

The key insight from FCN (2015): replace a classification network's final fully-connected layers with 1×1 convolutional layers. This changes the output from a single class vector to a spatial feature map of shape $H' \times W' \times K$ — one class score per pixel per class.

Problem: aggressive downsampling (max pooling) in the backbone reduces the spatial resolution by 32× for a standard VGG/ResNet backbone. FCN addresses this with transposed convolutions (sometimes called "deconvolution") to upsample back toward the original resolution.

Encoder-Decoder with skip connections: U-Net

U-Net (2015, medical imaging) introduced skip connections from encoder to decoder:

Encoder: Progressive downsampling (pooling) that captures semantic context at the cost of resolution.
Decoder: Progressive upsampling that recovers spatial resolution.
Skip connections: Directly concatenate encoder feature maps into the corresponding decoder layer, preserving fine-grained spatial detail (exact boundaries) that would otherwise be lost through the encoding bottleneck.

U-Net remains the dominant architecture for tasks requiring precise boundary delineation.

Dilated/Atrous convolutions

A dilated convolution with dilation rate $d$ inserts $(d-1)$ zeros between kernel elements, expanding the receptive field without increasing parameters or reducing resolution. A 3×3 kernel with dilation rate 2 covers a 7×7 area.

This allows the network to capture large-scale context (necessary for semantic understanding) without the resolution loss of max pooling, making it central to architectures like DeepLab.

Evaluation: mean Intersection over Union

$\text{mIoU} = \frac{1}{K}\sum_{k=1}^{K} \frac{TP_k}{TP_k + FP_k + FN_k}$

For each class $k$ , compute IoU between predicted and ground-truth pixel masks. Mean IoU averages across all $K$ classes, treating all classes equally regardless of how many pixels they occupy.

Code

1import torch2from torchvision import models3from torchvision.transforms import functional as F4from PIL import Image5import numpy as np6
7# Load DeepLabV3+ with ResNet-50 backbone (pre-trained on COCO + VOC)8model = models.segmentation.deeplabv3_resnet50(weights="DEFAULT")9model.eval()10
11# Load and preprocess image12image = Image.open("street.jpg").convert("RGB")13original_size = image.size  # (W, H)14
15input_tensor = F.to_tensor(image).unsqueeze(0)16input_tensor = F.normalize(17    input_tensor,18    mean=[0.485, 0.456, 0.406],19    std=[0.229, 0.224, 0.225]20)21
22with torch.no_grad():23    output = model(input_tensor)["out"]  # [1, num_classes, H, W]24
25# Predicted class per pixel26predicted_mask = output.argmax(dim=1).squeeze().numpy()  # [H, W]27
28# PASCAL VOC class labels (20 classes + background)29VOC_CLASSES = [30    "background", "aeroplane", "bicycle", "bird", "boat",31    "bottle", "bus", "car", "cat", "chair", "cow",32    "diningtable", "dog", "horse", "motorbike", "person",33    "pottedplant", "sheep", "sofa", "train", "tvmonitor"34]35
36# Count pixels per class37for class_id in np.unique(predicted_mask):38    pixel_count = (predicted_mask == class_id).sum()39    pct = 100 * pixel_count / predicted_mask.size40    print(f"  {VOC_CLASSES[class_id]:<15}: {pixel_count:>8} px ({pct:.1f}%)")

1# Compute mIoU for a batch (training evaluation)2def compute_mean_iou(preds, targets, num_classes):3    """4    preds: [B, H, W] predicted class per pixel5    targets: [B, H, W] ground truth class per pixel6    """7    iou_list = []8    for cls in range(num_classes):9        pred_cls = (preds == cls)10        tgt_cls = (targets == cls)11        intersection = (pred_cls & tgt_cls).sum().float()12        union = (pred_cls | tgt_cls).sum().float()13        if union == 0:14            continue  # class not present, skip15        iou_list.append(intersection / union)16    return torch.stack(iou_list).mean().item()

Watch Out For

Class imbalance in pixel counts

Sky and road can account for 80% of pixels in a driving scene; pedestrians may be < 1%. Standard cross-entropy loss trains the network to optimize dominant class accuracy at the expense of rare, often more important classes. Use class-weighted loss, focal loss, or oversample scenes with rare classes.

Confusing semantic and instance segmentation

Semantic segmentation labels all pixels of the same class identically — two people touching get the same "person" label. Instance segmentation (Mask R-CNN) labels each individual object instance separately. Choose the right task: semantic for scene understanding, instance for counting, tracking, or separating touching objects.

The Quick Version

Semantic segmentation assigns one class label to every pixel in an image.
FCN replaced FC layers with convolutions to produce spatial output; U-Net added skip connections for precise boundaries.
Dilated convolutions expand receptive fields without reducing resolution.
Evaluation: mean IoU (mIoU) across all classes — treats all classes equally.
Class imbalance (rare foreground vs. dominant background) is the main training challenge.