LVIS Long Tail Segmentation

A thousand-plus vocabulary with rare objects kept in, built to test whether segmenters handle the tail rather than the head.

LVIS keeps rare objects in the vocabulary, so frequent, common and rare splits score the tail separately

Why Does This Exist?

Most grounds let rare objects vanish into background, so models ace frequent classes and ignore the rest. LVIS built a thousand-plus vocabulary with federated labeling that keeps rare instances, then scores frequent, common and rare separately. It is the check for instance work that claims broad coverage.

This page covers purpose, federated I/O notes and tail scoring. Mask architectures live in Mask2Former.

Think of It Like This

A census that keeps rare surnames

Most lists drop surnames seen once. A full census keeps them and reports how well you spelled the common names versus the rare ones separately.

LVIS is that census. Head skill and tail skill get different grades on purpose.

How It Actually Works

Purpose is long-tail detection and segmentation. Input is one photo, outputs are boxes and masks over a broad vocabulary with some images only exhaustively labeled for a subset. Appropriate use is tail-aware benchmarking and repeat-factor sampling tests. Report AP for rare, common and frequent rather than one pooled number.

Toy tail walkthrough

Take AP scores 0.45 frequent, 0.30 common and 0.10 rare. The pooled headline near 0.35 hides the 0.10 tail. The gap $0.45 - 0.10 = 0.35$ is the long-tail penalty, and closing it is the actual task here.

Code

Split averaging that keeps the tail visible.

1import numpy as np2
3
4def tail_gap(ap_f: float, ap_r: float) -> float:5    return round(ap_f - ap_r, 3)  # head minus tail, the penalty to close6
7
8print(tail_gap(0.45, 0.10))9# -> 0.35

A 0.35 gap says head strength does not imply tail skill.

Watch Out For

Treating unlabeled as negative on federated images

Some images label only a subset exhaustively, so an unmarked object may simply be outside the labeled subset. The symptom is false negatives that punish correct predictions. Fix it by honoring the federated flags and negative sets instead of assuming full exhaustiveness.

The Quick Version

LVIS maps photos to boxes and masks over a thousand-plus vocabulary with rare kept in.
Input is a photo, outputs are federated boxes and masks with subset exhaustiveness.
Toy 0.45 head minus 0.10 tail gives a 0.35 long-tail gap.
Use it to test tail-aware detection and segmentation honestly.
Report rare, common and frequent separately.