Open-Set Recognition in Vision
Closed classifiers assign every image to a known class. Open-set recognition adds the honest answer: none of the above, with distance to prove it.
Why Does This Exist?
A 40-way product classifier meets its 41st category in the wild and must call it one of the 40. Softmax always sums to one, so something always wins, often confidently. Open-set recognition adds a rejection option: measure distance from the input to each known class in feature space, and answer unknown when everything is far. It is the deployment guardrail for every closed classifier on this site, including image classification.
The mechanism is cousin to prototypical networks: class means plus distances, now used to refuse rather than assign. Unknowns that look alike group into discovery queues for the next labeling round.
Think of It Like This
A bouncer with a guest list and a rope line
A club bouncer holds a list of 40 invited parties and a rope line at fixed distance from the door. Guests matching a party walk in; anyone standing beyond the rope from every party waits outside, no matter how confident they sound. The threshold is the rope position. The analogy stops at calibration: rope lines stay put all night, while distance scales shift with lighting and cameras and need re-tuning per site.
How It Actually Works
Embed the image, compute distance to each known-class prototype or model the per-class distance distribution from training, then reject when the nearest distance exceeds a threshold set on validation unknowns. Scoring variants use softmax confidence, energy, or reconstruction error, but distance in a good embedding beats confidence alone because unknown inputs routinely produce confident softmax winners.
Worked example with recomputed numbers
Forty known classes plus 100 held-out unknown images. Closed accuracy on knowns is 0.90. Set the threshold at the 95th percentile of known validation distances. It rejects 81 of the 100 unknowns while keeping known accuracy at 0.90, since 5 percent of knowns now also refuse. Tighten to the 90th percentile and rejections rise to 90 of 100 while known accuracy falls to 0.86. The two operating points price the trade: each 9 unknowns caught costs about 4 knowns refused here.
Threshold discipline
Choose thresholds on validation unknowns that resemble deployment novelty, not on toy noise. Recalibrate per site and per model update, because embedding scales move. Log every rejection with its distance for the labeling queue.
Watch Out For
Tuning the threshold on the test unknowns
Picking the rope position to maximize the reported rejection rate tests on the training set and inflates 81 toward 95. The symptom is a guardrail that fails on the next novel category. Split unknowns into threshold-setting and reporting sets, and report both numbers.
The Quick Version
- Closed softmax always picks a known class, so unknowns need an explicit reject path.
- Distance-to-prototype with a validation-set threshold rejects 81 of 100 unknowns here.
- Known accuracy stays 0.90 at the 95th-percentile rope position.
- Tightening catches more unknowns and refuses more knowns, priced directly.
- Recalibrate per site and split unknowns for tuning versus reporting.