Data Concepts

Active Learning

Labelling budget is finite, so spend it on the examples the model finds confusing. Random sampling spends most of it confirming what the model already knows.

Random sampling scatters a labelling budget across a pool where most examples are already easy, while uncertainty sampling spends the same budget on the narrow band around the decision boundary

Why Does This Exist?

You have 10,000 unlabelled documents and budget to label 500.

Pick them at random and most of your 500 will be examples the model already gets right. If the decision boundary only affects 8% of the pool, roughly 40 of your labels land somewhere that changes the model, and 460 confirm things it had already worked out.

Choose them by what the model is unsure about and all 500 can land in that band. Same budget, same annotators, a different question asked before the queue is built.

That's active learning: a loop rather than a technique. Train on what you have, score the unlabelled pool, label the most informative examples, retrain. It's one of the highest-leverage things available when labels are the constraint, which they usually are.

Think of It Like This

A driving instructor who only books motorway lessons

A learner is fine on quiet roads and terrible at roundabouts. Every lesson is an hour and there's money for ten.

Book ten random hours and you'll spend most of them driving down roads they can already handle. Everyone leaves satisfied — the lessons went well, because easy things go well — and the test still gets failed at the first roundabout.

A good instructor watches for hesitation and books the next hour there. Same ten hours, aimed at the places the learner can't yet decide.

One caution the analogy carries too. Only ever practising roundabouts produces someone who can't do a three-point turn, because nobody checked whether the rest of the skill set held. Uncertainty is a good guide and it isn't the whole curriculum.

How It Actually Works

The loop

Label a small seed set. Train. Score every unlabelled example by how much labelling it would help. Send the top $k$ to annotators. Retrain. Repeat.

The batch size matters: labelling one at a time is statistically ideal and operationally absurd, since annotators need a queue. Batches of a few hundred are the practical compromise, which is why diversity within a batch becomes a real concern below.

Ways to score usefulness

Least confidence takes one minus the top predicted probability. Cheap, and it only looks at the winning class.

Margin takes the gap between the top two probabilities, which is better for many classes — a three-way tie at 0.33 is more informative than a 0.4/0.35/0.25 split, and least confidence can't tell them apart.

Entropy uses the whole predicted distribution, so it's the right default for multi-class.

Query by committee trains several models and picks the examples they disagree on most. This is the strongest signal available, because disagreement between models captures uncertainty about the model rather than just uncertainty in one model's output. An ensemble, or several dropout passes, gives you a committee cheaply.

Expected model change asks which example would move the parameters most. Principled, expensive, rarely worth it in practice.

The two failure modes it invites

Redundancy within a batch. The 500 most uncertain examples are often 500 near-identical examples, because whatever confuses the model tends to confuse it in clusters. You pay for 500 labels and get the information of about 20.

Fix it by combining uncertainty with diversity: cluster the uncertain candidates and take a few from each cluster, or penalise similarity to what's already selected. Every practical system does some version of this.

Sampling bias in the labelled set. Your labelled data is now deliberately unrepresentative — concentrated at the boundary, by construction. So it's no longer a valid basis for estimating accuracy on the real distribution, and it can also skew the model, since the class balance of your labelled set drifts away from reality.

Two consequences worth being firm about. Keep a randomly-sampled held-out set, labelled separately, and never actively select into it — otherwise you have no honest estimate of anything. And keep some fraction of each batch random, typically 10 to 20%, both as a check on the strategy and to keep the labelled set from collapsing onto the boundary.

The cold start

With no labels, there's no model, so there's no uncertainty to sample by. The first batch has to come from somewhere else: random sampling, or diversity sampling — cluster the unlabelled pool by embedding and take representatives from each cluster, which gets coverage before you have a model. That's usually the better opening move.

Where it works and where it doesn't

It shines when unlabelled data is abundant, labelling is expensive, and the class of interest is rare — imbalanced problems are where the gains are largest, because random sampling of a 0.5% positive class is almost entirely wasted.

It's a poor fit when labelling is cheap enough to just label everything, when the model is retrained rarely so the loop can't turn, or when annotators are queued in a way that can't absorb a changing priority order.

Worked example

Five documents with predicted probabilities of 0.98, 0.52, 0.03, 0.61 and 0.49.

Least confidence is one minus the distance from the decision boundary: 0.02, 0.48, 0.03, 0.39, 0.49. Ranked, the most useful are document five, then two, then four. Documents one and three are near-certain — labelling them tells the model what it already believes.

Now the budget arithmetic. 500 labels from a pool where 8% of examples sit near the boundary gives roughly 40 informative labels under random sampling. Uncertainty sampling can put all 500 there, which is 12.5× more useful labelling from the same spend.

Worth stating the honest caveat: that 12.5 is an upper bound. Redundancy in the selected batch eats into it, so a realistic gain is more like 3 to 5×. Still the largest lever available when labels are the constraint.

Show Me the Code

1import numpy as np2
3p: np.ndarray = np.array([0.98, 0.52, 0.03, 0.61, 0.49])  # model probability of class 14budget: int = 5005near_boundary_share: float = 0.08                          # of the unlabelled pool6
7def least_confidence(probs: np.ndarray) -> np.ndarray:8    """How far the top class sits from certainty. Higher means more worth labelling."""9    return 1 - np.maximum(probs, 1 - probs)10
11scores: np.ndarray = least_confidence(p)12print(np.round(scores, 2).tolist())         # -> [0.02, 0.48, 0.03, 0.39, 0.49]13print(np.argsort(-scores)[:3].tolist())     # -> [4, 1, 3]  label these first14print(round(budget * near_boundary_share))  # -> 40   informative labels, sampling at random15print(round(budget / (budget * near_boundary_share), 1))  # -> 12.5  the upper bound on the gain

np.argsort(-scores) is the entire mechanism. Everything else on this page is about not trusting it blindly.

Watch Out For

Estimating accuracy on an actively-selected set

You run five rounds, end up with 2,500 labels, hold out 20% of them, and report 84%.

That holdout is drawn from a set you deliberately selected to be hard. It over-represents the boundary by construction, so the number is pessimistic — and unpredictably so, because how pessimistic depends on the strategy, the round, and how much redundancy crept in. It isn't an estimate of anything you can act on.

The mirror-image error is worse. If your acquisition function happened to favour a class, the labelled set's balance no longer matches production, so both your metric and your model's calibration are off, in a direction nobody has measured.

Label a random sample separately, before the loop starts, and ring-fence it. It's the only set that can answer "how good is this model", and it needs to be sized for the difference you care about, the same way any test set does. Spending 20% of a budget on a random evaluation set feels wasteful and is the price of knowing anything.

A batch of 500 near-identical uncertain examples

Rank the pool by uncertainty, take the top 500, send to annotators. The obvious implementation, and it wastes most of the batch.

Uncertainty clusters. If the model is confused by invoices from one particular vendor whose format it hasn't seen, the 500 most uncertain documents can be 500 invoices from that vendor. You pay 500 labels to learn one thing, the annotators get bored, and the next round finds a different cluster and does it again.

There's a second-order version: after the retrain, the model is now confident about that cluster and the next batch collapses onto the next cluster, so the loop oscillates between narrow regions instead of covering the space.

Combine uncertainty with diversity. Cluster the top few thousand candidates by embedding and take a handful from each cluster, or greedily penalise similarity to what's already in the batch. Then keep 10 to 20% of every batch randomly sampled, which bounds the damage when the acquisition function is wrong.

The Quick Version

Labels are usually the constraint, so which examples you label matters more than how many.
Random sampling spends the budget confirming what the model knows. If 8% of the pool is near the boundary, 500 random labels buy 40 informative ones.
The loop: seed, train, score the pool, label the top $k$ , retrain.
Least confidence is cheap, margin is better for many classes, entropy is the multi-class default, and a committee's disagreement is the strongest signal.
The 500 most uncertain examples are often near-identical. Combine uncertainty with diversity, or the batch buys one thing.
Never estimate accuracy on an actively-selected set. Label a random holdout first and ring-fence it.
Keep 10 to 20% of each batch random, as a check and to stop the labelled set collapsing onto the boundary.
Cold start has no model, so open with diversity sampling over embeddings rather than pure random.
Biggest gains on imbalanced problems. Not worth it when labelling is cheap or retraining is rare.