Panoptic-DeepLab Mergers

Panoptic-DeepLab predicts class maps, instance centres, and pixel offsets together, then assigns each pixel to its nearest centre for a clean merge.

Panoptic-DeepLab finds each object centre and pulls every thing pixel toward its own centre to form instances.

Why Does This Exist?

Panoptic FPN needs boxes, RoI alignment, and a rulebook to merge two heads. Panoptic-DeepLab (Cheng et al., 2020) goes box-free: three dense heads on a shared backbone predict semantics, a centre heatmap for things, and a 2D offset from every thing pixel to its centre. Grouping is nearest-centre assignment. One backbone, three maps, no proposals, friendly to mobile budgets.

Think of It Like This

Shepherds calling scattered sheep

Each flock's shepherd lights a beacon (centre heatmap peak). Every sheep hears directions to its own beacon (offset vectors) and walks there. Sheep in open pasture (stuff) ignore beacons and stay classified by field. Counting flocks means counting beacons, not building fences first.

Where it stops: two beacons lit shoulder to shoulder in a dense crowd merge into one glow, and flocks blend.

How It Actually Works

The backbone (often Xception or MobileNet with atrous convolutions from the DeepLab line) feeds three heads: semantic logits over $K$ classes, a single-channel centre heatmap trained with MSE against Gaussian blobs, and two offset channels trained with L1 only on thing pixels. Inference applies keypoint NMS to centres, shifts each thing pixel by its offset, and assigns it to the nearest surviving centre. Majority vote from the semantic head labels each instance; stuff pixels keep dense labels.

Worked example

Centres at $(50, 60)$ and $(90, 60)$ . A pixel at $(70, 62)$ predicts offset $(-19, -2)$ , landing at $(51, 60)$ : distance $1$ to centre one, $39$ to centre two, so it joins instance one. A pixel predicting offset $(+21, -1)$ lands at $(91, 61)$ and joins instance two. One mis-set offset of $(0, 0)$ would leave the pixel mid-way and risk a wrong assignment, which is why offset L1 on edges matters.

Code

1import math2
3# Nearest-centre assignment after offset shift.4pixel, offset = (70, 62), (-19, -2)5landed = (pixel[0] + offset[0], pixel[1] + offset[1])6centres = [(50, 60), (90, 60)]7dists = [math.dist(landed, c) for c in centres]8print(landed, [round(d, 1) for d in dists])9# -> ((51, 60), [1.0, 39.0])

Watch Out For

Centre collisions in crowds

Adjacent people produce one merged heatmap blob after NMS. Symptom: undercounted crowds. Fix: raise heatmap resolution, shrink Gaussian sigma, and tune keypoint NMS radius on crowd validation slices.

Offset noise on far pixels

Distant rim pixels regress long offsets with high variance. Symptom: fringed instance borders. Fix: weight offset loss toward near-centre pixels and let the semantic head own the far rim.

The Quick Version

Three heads: semantics, thing-centre heatmap, per-pixel centre offsets.
Pixels join the nearest surviving centre after shifting by their offsets.
No boxes, no RoI steps, no fusion rulebook.
Runs well on efficient backbones for on-device panoptic maps.
Crowded centres and long offsets are the accuracy limits.