Long-Tailed Recognition in Vision
Real image collections give a few classes thousands of examples and hundreds of classes a handful. Standard training then serves the head and starves the tail.
Why Does This Exist?
Street scenes, species catalogs, and product feeds all follow the same shape: a handful of common classes own most images while hundreds of rare ones own five each. Plain training minimizes average loss, which means pleasing the head classes and ignoring the tail. Overall accuracy looks fine at 0.84 while tail accuracy sits at 0.31, and the tail is where the errors cost most.
The repair has two stages: learn features from the natural distribution, then retrain the classifier on balanced sampling. Report head, medium, and tail separately or the mean hides the failure. Sibling guardrails for unknown inputs are open-set recognition, built on image classification.
Think of It Like This
An election where cities outvote villages
Counting raw votes lets three big cities decide every rural road budget. Balanced representation gives each district delegates so villages keep a voice without silencing cities. Resampling is delegate allocation for classes. The analogy stops at features: delegates vote independently, while tail classes borrow the features learned from head-class abundance.
How It Actually Works
Stage one trains the backbone on the natural long-tailed stream so features see maximum data. Stage two freezes the backbone and retrains only the classifier with class-balanced sampling, so each class contributes equally to the decision boundaries. Re-weighting losses and margin adjustments that favor rare classes add further gains. Augmentation targeted at tail classes multiplies the effect of their few images.
Worked example with recomputed numbers
One thousand classes: 10 head classes with 5,000 images each, 100 medium with 200 each, 890 tail with 5 each. Plain training scores 0.84 overall with head 0.95, medium 0.71, tail 0.31. Freeze the backbone and retrain the classifier class-balanced: overall moves to 0.79 while tail climbs to 0.58, medium to 0.74, head eases to 0.93. Overall falls 0.05 while tail nearly doubles, which is the trade stated per split rather than hidden in the mean.
Sampling detail that matters
Repeat-factor sampling that oversamples tail images per epoch beats loss re-weighting alone here, because rare images need repeated views with different augmentation. Cap repeats so five tail images do not memorize: pair repeats with strong augmentation and stop on tail validation, not training loss.
Watch Out For
Reporting only overall accuracy
Overall 0.84 certifies a model with 0.31 tail accuracy for deployment, and tail users pay for the headline. The symptom is a launch that fails exactly on rare but important cases. Report head, medium, and tail splits at every checkpoint and gate releases on tail floors.
The Quick Version
- Real collections starve the tail: 5 images per tail class against 5,000 head.
- Plain training hides it: 0.84 overall with tail at 0.31.
- Balanced classifier retraining lifts tail to 0.58, medium holds 0.74, head 0.93.
- Overall dips to 0.79, which is the honest price of serving rare classes.
- Always report per-split accuracy and gate on the tail floor.