Active Learning Pipelines

Active learning pipelines spend each labeling dollar on the images the model finds most confusing, then retrain and repeat.

Fifty thousand unlabeled images loop through uncertainty scoring that picks 500 for labeling and lifts accuracy from 0.71 to 0.78

Why Does This Exist?

Labeling every street image costs a fortune, and most of them teach nothing: mile after mile of empty road. Random sampling spends the budget on duplicates while rare corners stay uncovered. Accuracy plateaus even as labeling bills grow.

Active learning pipelines flip the order. Train on a seed set, score the unlabeled pool for confusion, label the top slice, and retrain. The generic strategy lives in active learning, dataset craft lives in dataset preparation, and low-data regimes live in few-shot vision.

Think of It Like This

A tutor who assigns only missed problems

A good tutor does not reassign the whole textbook. They mark a quiz, find the problems you missed, and assign those plus close variants next.

The model is the student, entropy is the marking, and the labeling budget is homework time. The analogy stops at forgetting: students retain, while retrained models can slide on old slices unless you keep them in the mix.

How It Actually Works

Each round trains the detector on labeled images, then scores every unlabeled image with an acquisition function. Least-confidence picks low top scores, entropy picks flat distributions, and diversity methods add spread so the batch is not 500 near-duplicates of one corner case. Humans label the picks, the labeled set grows, and the loop repeats until the budget or a target metric hits.

Worked example

Pool: $50{,}000$ unlabeled street frames, seed model at $0.71$ accuracy. Entropy over three class probabilities picks the top $500$ : for $[0.34, 0.33, 0.33]$ entropy is about $1.10$ , while $[0.90, 0.05, 0.05]$ gives about $0.39$ , so flat outputs win. Label those $500$ , retrain, accuracy reaches $0.78$ . Same $500$ -image spend on random frames historically gave $0.73$ , so selection bought $0.05$ extra for free.

Code

Entropy ranking over a small pool, the core of the query step.

1import math2
3
4def entropy(probs: list[float]) -> float:5    """Shannon entropy in nats; flat distributions score high."""6    return -sum(p * math.log(p) for p in probs if p > 0)  # guard log(0)7
8
9pool = [[0.34, 0.33, 0.33], [0.9, 0.05, 0.05], [0.5, 0.4, 0.1]]10print([round(entropy(p), 2) for p in pool])  # -> [1.1, 0.39, 0.94]

Watch Out For

Querying near-duplicates of one confusion

Pure uncertainty loves a crowd: 500 frames of the same foggy intersection top the list together. Labels repeat one lesson while other confusions starve.

Add diversity: cluster embeddings and cap picks per cluster. Check the pick map before paying for labels.

Letting the seed bias compound

Early models fear rare classes, so they never query them, so they never learn them. The loop then perfects common scenes and ignores the tail.

Seed with stratified coverage and track per-class queries. Force a quota for classes with zero picks after two rounds.

The Quick Version

Active learning loops train, score confusion, label the top slice, and retrain.
Entropy, least-confidence, and diversity sampling form the standard query set.
Worked numbers: $500$ of $50{,}000$ labeled, $0.71$ to $0.78$ accuracy.
Random $500$ reached only $0.73$ , so selection added $0.05$ .
Cap picks per cluster and quota rare classes or bias compounds.