Encoder-Decoder Segmentation Architectures

A downsampling encoder reads what is in the image while an upsampling decoder recovers where it is, and skip bridges return the edges pooling lost.

A 256 wide image shrinks to a 64 wide bottleneck for meaning, then grows back to 256 while skip bridges return the lost edges for a sharp mask

Why Does This Exist?

Classifiers compress: each pooling step halves the spatial map while doubling what the network knows, ending in one label for the whole image. That trade is perfect for naming and fatal for outlining. Surgery planning, street scenes, and cell counting need the opposite output, one label per pixel at full resolution, which is what semantic segmentation means.

Running the classifier backward is the fix. An encoder downsamples to learn what is present, a decoder upsamples back to full size to say where, and skip bridges smuggle the fine edges across before pooling destroys them. U-Net is the canonical shape: symmetric down and up paths with a copy-and-concatenate bridge at every level. The upsampling operator itself is taught in transposed convolution, and the dilated alternative that skips the decoder is the DeepLab family.

The one idea you need first

Downsampling by pooling or strided convolution halves width and height while growing channels, trading location for meaning. Upsampling doubles the map back, either learned through transposed convolution or fixed through interpolation plus a convolution. Channels are the stacked feature maps at each stage: more channels mean richer descriptions per location. A skip connection copies a map from one stage to a later one, bypassing whatever sits between.

Think of It Like This

A surveyor and a cartographer sharing field notes

A surveyor climbs a hill to read the whole valley: river, road, village. From that height the river is a blue thread with no banks. Meanwhile a cartographer must draw every bank at street scale. The surveyor shouts down the summary while runners carry the early close-up sketches straight to the drawing table.

The encoder is the climb, the decoder is the drawing, and the runners are the skip bridges: without them the map shows a river with guessed banks. It stops holding at instances: the map paints every water pixel blue but cannot say which droplet belongs to which of two touching lakes, and neither can U-Net split two touching cells of one class.

How It Actually Works

Three mechanisms work in tandem. The encoder repeats convolution plus pooling: each stage halves width and height and roughly doubles channels, so a $256 \times 256$ image with 16 channels becomes $128 \times 128$ with 32, then $64 \times 64$ with 64 at the bottleneck, trading location for meaning. The decoder mirrors it with upsampling plus convolution: $64 \times 64$ grows to $128 \times 128$ and back to $256 \times 256$ , relearning location. The skip bridges copy each encoder map across before its pooling and concatenate it with the matching decoder map, so deep meaning meets crisp location at every scale before the final $1 \times 1$ convolution votes per pixel.

Worked example

Trace one $256 \times 256$ image with a single channel through two pooling stages. After the first encoder block and pool the map is $128 \times 128$ with 32 channels; after the second it is $64 \times 64$ with 64 channels at the bottleneck, which holds $64 \times 64 \times 64 = 262{,}144$ values. The first upsampling returns to $128 \times 128$ with 32 channels, and concatenating the 32-channel skip from the matching encoder stage makes $32 + 32 = 64$ channels entering the decoder convolution. The second upsampling returns to $256 \times 256$ with 16 channels, plus the 16-channel skip makes 32, and a final $1 \times 1$ convolution maps those to 2 class scores per pixel: $256 \times 256 \times 2 = 131{,}072$ outputs. Every doubling is exact because 256 divides by $2^2 = 4$ .

Code

The snippet builds a one-pool U-Net and prints the shapes that prove the bridge arithmetic: the skip and the upsampled map share a size, and the merge doubles the channels.

1import torch2import torch.nn as nn3
4
5class TinyUNet(nn.Module):6    """One-pool U-Net: encoder, bottleneck, decoder with one skip bridge."""7
8    def __init__(self) -> None:9        super().__init__()10        self.enc = nn.Conv2d(1, 16, kernel_size=3, padding=1)11        self.pool = nn.MaxPool2d(2)12        self.bottle = nn.Conv2d(16, 32, kernel_size=3, padding=1)13        self.up = nn.ConvTranspose2d(32, 16, kernel_size=2, stride=2)14        self.dec = nn.Conv2d(32, 16, kernel_size=3, padding=1)15        self.out = nn.Conv2d(16, 2, kernel_size=1)16
17    def forward(self, x: torch.Tensor) -> torch.Tensor:18        skip = torch.relu(self.enc(x))  # full resolution edge detail19        deep = torch.relu(self.bottle(self.pool(skip)))20        up = self.up(deep)  # back to full resolution21        merged = torch.cat([up, skip], dim=1)  # 16 + 16 = 32 channels22        return self.out(torch.relu(self.dec(merged)))23
24
25net = TinyUNet()26logits = net(torch.randn(1, 1, 64, 64))27print(logits.shape)28# -> torch.Size([1, 2, 64, 64])29print(net.up.weight.shape)30# -> torch.Size([32, 16, 2, 2])

The output is 2 class scores at every one of the $64 \times 64$ pixels, the dense prediction the task demands. The transpose weight shape reads as 32 inputs and 16 outputs because transposed convolution stores its weights transposed relative to the direction data flows.

Watch Out For

Odd input sizes that break the bridge concatenation

Two pools need widths divisible by 4, three pools by 8. A 100-wide image pooled twice gives 25, and upsampling 25 twice returns 100 only if every rounding agrees; one floor too many and the decoder map is 99 or 101 against a 100-wide skip, which throws a shape mismatch on concatenation. Pad inputs to multiples of $2^N$ for $N$ pools, or crop the skip to the upsampled map before concatenating.

Upsampling without bridges and calling it U-Net

A decoder alone rebuilds size but invents edges: boundaries come out blobby and the boundary overlap score lags the pixel accuracy by a wide margin. The symptom is masks that look right at thumbnail scale and wrong zoomed in. Check that each decoder stage actually concatenates its skip; a missing bridge is one deleted line that halves edge quality.

The Quick Version

Classifiers trade location for meaning; segmentation needs both at full resolution.
The encoder halves width while doubling channels down to a 64-wide bottleneck in the worked example.
The decoder doubles back up, and each skip concatenation restores the edges pooling lost.
Concatenation arithmetic is exact: 32 upsampled plus 32 skip channels enter the decoder convolution.
Input widths must divide by 2 per pool or the bridges mismatch.
U-Net paints every pixel but cannot split two touching shapes of one class.