Weak Supervision

Nobody will hand-label two million rows. Write a dozen noisy rules instead, then model how often each rule errs and combine them into one soft label.

Three rules at 80, 70 and 60 percent feed a label model that weights each vote by estimated accuracy, so the soft label beats a flat majority that scores 0.788

Why Does This Exist?

You need labels for two million tickets. Hand-labelling them takes a year and a budget no one approves. Active learning spends a small budget well, but you still need volume.

So you write rules. If the text holds an unsubscribe link, call it marketing. If the sender domain sits on your internal list, call it internal. If an invoice pattern appears, call it billing. Each rule is wrong often, none covers everything, and they disagree.

That sounds useless, and the useful part is this. If you can estimate how often each rule errs, you can combine the votes into a soft label that beats any single rule. You moved human work from labelling rows to writing rules, and one rule labels a million rows.

This page covers rule votes turned into training labels. How careful human labels get made is labelling and annotation. How you spend a small labelling budget on the hardest rows is active learning.

Basics first: a label is a human judgement paired with an input, and agreement between two annotators sets the ceiling a model can reach. If that ceiling is new, read labelling and annotation first. Everything below assumes you know why 500 double-labelled rows can save a project.

Think of It Like This

Three neighbours guessing rain

Three neighbours tell you each morning whether it will rain. One is right 8 days in 10, one 7 in 10, one 6 in 10.

Count plain votes and you beat the two weaker neighbours. You still lose to the best one alone, because two weak voices outvote one good voice when every voice counts the same.

Now imagine you kept score for a year, so you know each hit rate. You would listen most to the 8 in 10 neighbour, and let the 6 in 10 voice break ties only. That weighting is the whole trick.

The part that feels like magic is that you do not need the scorebook. How often the three agree with each other already reveals roughly who is best. The neighbour who sides with the others constantly is probably good. The one who stands alone often is probably noise. No ground truth required.

Where it stops: neighbours watch the same sky, so a shared blind spot fools all three at once. Correlated rules fail the same way, and agreement then reads as proof when it is only repetition.

How It Actually Works

Labelling functions that may abstain

A labelling function takes a row and returns a label or abstains. The abstain matters. A keyword rule should fire only where it has an opinion. Forcing it to guess everywhere trades its precision away.

Sources vary. Keyword and pattern matches. A heuristic over metadata. An older knowledge base matched by name, which the field once called distant supervision. A small off-the-shelf model. A prompted large model, now the most common source: broad coverage, cheap, and wrong in correlated ways.

Describe each function by two numbers. Coverage is the share of rows it fires on. Accuracy is how often it is right when it fires. You want complementary coverage and you accept middling accuracy.

The label model, and why flat votes lose

Stack the votes into a matrix: rows are examples, columns are functions, cells hold a label or an abstain. A label model estimates each function accuracy from that matrix alone, then emits one soft label per row.

It works from agreement structure. Two functions that agree far more often than chance are probably both accurate. One that disagrees with everything is probably noise. That estimation is the technical core, and it needs no ground truth.

A flat majority is the special case where every function counts the same. The worked example below shows what that costs when quality is uneven. Train your end model on the soft labels with a noise-aware loss, weighting each row by its confidence. The end model then sees raw text the rules never saw, so it can generalise past them.

Correlated rules break the math

The estimation assumes errors land roughly independently. Two keyword lists with heavy overlap, or three prompts to the same model with different wording, violate that. The label model reads their constant agreement as mutual proof and inflates both accuracies.

The output gives no warning. Confidence looks high, coverage looks healthy, and errors concentrate exactly where the correlated block fires. Declare known correlations where your tool allows it, or merge near duplicates before you trust the weights.

Volume here, truth elsewhere

Weak supervision buys volume. Human labels still buy truth. Hold out a few hundred careful human labels for evaluation, because a test set built from the same rules only measures whether the rules agree with themselves. The loop that works is write rules, fit the label model, inspect conflicts and low confidence rows, add a rule for that region, repeat.

Worked example

Three functions, all firing, accuracies 70, 80 and 60 percent, erring independently. A majority is right when at least two are right.

All three right: $0.7 \times 0.8 \times 0.6 = 0.336$ . Exactly two right: $0.7 \times 0.8 \times 0.4 = 0.224$ , plus $0.7 \times 0.2 \times 0.6 = 0.084$ , plus $0.3 \times 0.8 \times 0.6 = 0.144$ , which sums to $0.452$ . Total $0.336 + 0.452 = 0.788$ . That beats the 70 and the 60 rules, and loses to the 80 rule alone.

Make all three 70 percent and the same vote scores $0.784$ , now above every member. Same mechanism, opposite verdict. The only change is how evenly accuracy spreads.

Push it to one expert at 90 with two coin flips at 55 each. The majority scores $0.748$ , well below the 90 rule. Two weak voices reliably outvote one good one, which is why weighting matters.

Code

1import numpy as np2from itertools import product3
4def majority_accuracy(accs: tuple[float, ...]) -> float:5    """Chance a flat vote is right if rules err independently."""6    total = 0.07    for outcome in product([True, False], repeat=len(accs)):8        p = np.prod([a if ok else 1 - a for a, ok in zip(accs, outcome)])9        if sum(outcome) * 2 > len(accs):10            total += float(p)11    return total12
13print(round(majority_accuracy((0.7, 0.7, 0.7)), 4))  # -> 0.78414print(round(majority_accuracy((0.7, 0.8, 0.6)), 4))  # -> 0.78815print(round(majority_accuracy((0.9, 0.55, 0.55)), 4))  # -> 0.748

Three calls, three verdicts on one mechanism. A label model exists so the third case stops happening.

Watch Out For

Rules that are secretly one rule

You write twelve functions and four share most keywords, or three prompt the same model with different wording.

Agreement between those four runs near 0.9, not because they are right but because they are the same guess in four coats. The label model reads that as four accurate witnesses and weighs them heavily, so wherever they err together the soft label is confidently wrong.

Nothing in the scores flags it. Check pairwise agreement before fitting and merge pairs above about 0.9, or declare the link if your tool supports it. Then slice accuracy on your human test set by which functions fired, because an aggregate number hides the region the block owns.

A test set built from the same rules

You hold out 10 percent of the soft labels for evaluation. That split measures nothing.

Those held-out rows came from the same functions, so a model that copies your regexes scores 94 percent while learning every blind spot they hold. An end model that beats the label model on that split can mean generalisation, or just smoother imitation. No rule-made split can tell them apart.

Hand-label a real test set before writing rules. A few hundred careful rows with a measured agreement rate is enough, and it is the only number that means anything. Keep it apart from training, as any honest split requires.

The Quick Version

Move human work from labelling rows to writing rules, because one rule labels a million rows.
A labelling function returns a label or abstains. Forcing a guess everywhere destroys precision.
Describe each rule by coverage and accuracy. Aim for complementary coverage and accept middling accuracy.
A label model estimates accuracy from agreement alone, then emits one soft label per row.
Flat votes assume equal quality. Three rules at 70 percent vote to 0.784, at 70 with 80 and 60 they vote to 0.788, at 90 with two 55s they vote to 0.748.
Train the end model on soft labels with a noise-aware loss so it can pass the rules.
Correlated rules inflate each other and produce confident errors. Check pairwise agreement near 0.9.
Prompted models are labelling functions with broad coverage and correlated errors.
Evaluation must be human-labelled. A rule-made holdout only grades imitation.