Instance Segmentation Splits Same-Class Masks

Instance segmentation gives every object its own pixel mask, so two overlapping people stay two instances instead of melting into one blob.

Semantic maps paint one class blob while instance heads pair each detected box with its own mask, so two overlapping people stay two counted objects

Why Does This Exist?

Bounding boxes are a crude approximation of shape: a box around a walking person includes half the sidewalk. Semantic segmentation fixes shape by classifying every pixel, but it groups all same-class pixels together, so two overlapping people melt into one person blob that cannot be counted or grasped.

Instance segmentation bridges the gap with both identity and shape: each detected object gets its own mask, as built by Mask R-CNN. Autonomous driving needs the exact outline of each pedestrian, and robotic grasping needs one specific item out of a bin of identical overlapping ones. Where every pixel needs both a class and an owner, the fuller accounting lives in panoptic segmentation.

Think of It Like This

One highlighter versus one pen per apple

A bowl of overlapping apples can be marked two ways. Semantic segmentation is one red highlighter over all apple pixels: the apple area is known but the count is lost where they overlap. Instance segmentation hands each apple its own pen, so apple one is blue, apple two is green, and outlines plus counts survive the pile.

The pens are the per-object masks and the bowl is the image. Where it stops: handing out pens costs a detection pass first, so a scene with hundreds of tiny overlapping instances pays far more compute than one highlighter sweep ever would.

How It Actually Works

The classic route is detect then segment. A backbone extracts feature maps and a region proposal network suggests boxes, then RoIAlign crops each box with bilinear interpolation so pixel locations survive exactly. One head predicts the class and refines the box while a parallel fully convolutional head predicts a binary mask per box, with the class head supplying the label so the mask head only decides shape.

Single-stage models skip proposals for speed: they predict whole-image prototype masks plus per-object coefficients and combine them linearly. The trade is real-time rates against softer edges on crowded scenes.

Worked example

Two overlapping people are predicted as masks of $100$ pixels each sharing $85$ pixels. The union is $100 + 100 - 85 = 115$ pixels. Mask IoU is $85 / 115 \approx 0.739$ , about $0.74$ : high overlap, yet the two masks stay separate instances with their own boxes and IDs instead of one blob.

Score the first mask against truth of $100$ pixels sharing $80$ : union is $100 + 100 - 80 = 120$ , IoU is $80 / 120 \approx 0.667$ . The number says good overlap; the instance IDs say two people, which is the part semantic maps cannot say.

Code

The point of this snippet is mask overlap arithmetic, not a network: pure standard-library code recomputes the worked example and shows why sharing one high IoU still leaves two instances.

1def mask_iou(shared: int, a: int, b: int) -> float:2    union = a + b - shared3    return round(shared / union, 3)4
5print(mask_iou(85, 100, 100))6# -> 0.7397print(mask_iou(80, 100, 100))8# -> 0.667

Two masks sharing $85$ of $100$ pixels each score about $0.74$ yet remain two instances. Overlap measures agreement; the per-object IDs carry the counting.

Watch Out For

Overlapping masks with no merge rule

Each object gets its own mask, so neighbors overlap on the same pixels and a flat label map cannot hold both. The symptom is flickering IDs or double-counted pixels at every occlusion boundary. Fix it with an explicit merge rule such as highest-score-wins or depth ordering before rendering one map.

Full-resolution masks for hundreds of proposals

High-resolution masks for hundreds of boxes eat VRAM fast, and training dies on the largest scenes first. The symptom is out-of-memory errors on crowded images while sparse ones train fine. Fix it by predicting small masks such as $28 \times 28$ and upsampling, or by moving to single-stage models when speed matters more than crisp edges.

The Quick Version

Instance segmentation pairs each detected box with its own pixel mask.
Two overlapping people stay two instances instead of one semantic blob.
The fixture scores $85 / 115 \approx 0.74$ mask IoU across the pair.
RoIAlign keeps pixel alignment exact; single-stage models trade edges for speed.
Overlapping outputs need a merge rule, and full-resolution masks need a memory budget.