Real-Time Segmentation

Real-time segmentation labels every pixel at camera rate by spending compute where masks need it and skipping what eyes will never check.

A 28 ms budget at 960 pixels keeps road edges while full resolution at 96 ms misses the frame slot

Why Does This Exist?

Driving, AR, and robots decide per pixel at $30$ frames per second. A $96$ ms mask arrives three frames late and steers from the past. Heavy decoders that win leaderboards miss the slot.

Real-time segmentation exists to hit the slot with useful masks. Input is video frames. Output is per pixel labels inside $33$ ms. The accuracy base is semantic segmentation; the object variant is instance segmentation. The medical detail keeper is U-Net.

Use it when latency binds safety or comfort. Use heavy offline models when every small object matters and time does not.

Think of It Like This

Sketching traffic, not painting portraits

A courtroom sketch artist catches pose and layout in seconds with few strokes, skipping pores the jury never needs. A portrait painter needs hours for the same face.

Real-time nets sketch: thin backbones, early downscale, light decoders. The analogy stops at reuse: the sketcher redraws each face while the net reuses backbone features across tasks, which is where the speed truly comes from.

How It Actually Works

Cut where eyes will not check: run at $960$ instead of $1920$ width, thin the backbone, share one encoder for all classes, and decode with few up steps plus skip adds. Contract is frame in, mask out within $33$ ms. A $960$ width run takes $28$ ms and holds road edges within $2$ pixels of the heavy mask. Full $1920$ width takes $96$ ms and misses three slots. Small poles under $6$ pixels wide fade first, which the spec must accept or handle with a second sensor.

Worked example

A street frame at $1920 \times 1080$ segments in $96$ ms on the test board. Dropping to $960 \times 540$ cuts pixels by $4$ times and time to $28$ ms, inside the $33$ ms slot for $30$ fps. Road edge shifts $1.8$ pixels average against the heavy mask, inside a $2$ pixel tolerance. Pole recall drops from $0.81$ to $0.64$ , the documented cost of speed that the safety case carries explicitly.

Watch Out For

Downscale that erases the class you need

Poles, wires, and distant pedestrians live under $8$ pixels wide. Halving resolution deletes them before the net runs. The symptom is clean road scores plus missing poles. Fix it with resolution aware specs, crop branches for far fields, or a second detector for thin classes.

Board numbers sold as product numbers

A demo GPU hits $28$ ms while the shipped chip needs $52$ ms with heat throttling. The symptom is green lab charts plus late frames in sun. Fix it by budgeting on the coldest shipped hardware with full camera pipelines running.

The Quick Version

Real-time masks must land inside $33$ ms for $30$ fps video.
Thin shared backbones plus light decoders buy the slot.
The fixture runs $28$ ms at $960$ width against $96$ ms at full width.
Road edges hold within $2$ pixels while thin poles pay the cost.
Spec shipped hardware and thin class handling before claiming real time.