Data Concepts

Labelling and Annotation

Your labels are the ceiling on your model. If two annotators disagree on a fifth of the examples, nothing you train can be more accurate than they are.

Two annotators agreeing on three quarters of a hundred items sounds respectable until you subtract the agreement they would have reached by guessing, which leaves a kappa of 0.49

Why Does This Exist?

A team spends four months and a large budget labelling 80,000 support tickets by intent. The model tops out at 71% and every attempt to improve it fails.

Then somebody double-labels 500 tickets and finds the annotators agree on 74% of them. The model was never going to beat 71%, because the labels are the ceiling. It had already learned everything the data could teach and was now fitting the disagreement.

That's the fact this page exists for. A supervised model approximates its labels, not the truth, so label quality is an upper bound on model quality rather than one input among many. And measuring that bound costs a fraction of what generating the labels cost — 500 double-labelled examples would have saved most of those four months.

Think of It Like This

Two examiners marking the same essays

A school has two examiners mark the same hundred essays independently, out of a pass or fail.

They agree on 75. On the other 25 one says pass and the other fail, which is a bit awkward and gets resolved by averaging.

Now the interesting question: what is a "correct" mark for those 25? There isn't one. The essays sit in a region where the criteria don't decide, and no amount of averaging creates an answer — it just picks one and hides the ambiguity.

If you now train a new examiner on these marks, the best they can do is reproduce the confident 75 and guess on the rest. Their ceiling was set before they saw an essay, by how well the criteria were written.

Rewriting the criteria is the only thing that raises it.

How It Actually Works

Guidelines are the product, not the labels

The labels are output. The guidelines are the artefact that determines their quality, and they're written iteratively rather than up front.

The loop that works: draft the guidelines, have two or three people label the same 100 examples, measure agreement, and read every disagreement. Each disagreement is either an annotator error or a gap in the guidelines, and it's usually the second. Patch the guidelines with the specific edge case and its ruling, then repeat.

Three rounds of that on 300 examples will do more for a model than 30,000 extra labels under vague guidelines. Guidelines that don't name edge cases are guidelines that let each annotator invent their own.

Measuring agreement properly

Raw agreement is the fraction of items two annotators labelled identically, and it flatters you badly on imbalanced tasks. If 95% of items are negative, two annotators who both say "negative" to everything agree 95% of the time and have told you nothing.

Cohen's kappa subtracts the agreement you'd expect from chance, given each annotator's own base rate:

\kappa = \frac{p_o - p_e}{1 - p_e}

where $p_o$ is observed agreement and $p_e$ is the agreement expected if both labelled independently at their own rates. Zero means chance-level; one means perfect. Rough reading: below 0.4 the task isn't defined, 0.4 to 0.6 is shaky, 0.6 to 0.8 is usable, above 0.8 is good.

Fleiss' kappa extends it to more than two annotators. Krippendorff's alpha handles ordinal and missing data. For spans and bounding boxes, agreement means overlap, so intersection-over-union with a threshold does the job.

Structures that raise quality

Multiple annotators plus adjudication. Two label, a third resolves conflicts. Expensive, and the resolution log is a gold mine for guideline fixes.

A gold set. A few hundred items labelled carefully by an expert, mixed invisibly into every annotator's queue as a continuous accuracy measurement per annotator. Without one you can't distinguish a hard task from a struggling annotator.

Label the boundary, not the bulk. Once agreement on obvious cases is high, extra obvious cases add nothing — that's active learning.

Don't discard the disagreement. The items two annotators split on are genuinely ambiguous, and that's information. Keeping a soft label, or a "disputed" flag, is more honest than a coin flip and lets you exclude them from evaluation, where an ambiguous item just adds noise to your metric.

Label noise, and what it does

Random noise mostly costs you sample efficiency: the model averages over it, given enough data. Systematic noise is the dangerous kind — one annotator who consistently misreads a category, or a guideline ambiguity that always resolves the same way — because the model learns the bias faithfully and no amount of data corrects it.

Which is why per-annotator accuracy on a gold set matters more than aggregate agreement. Aggregate agreement can look fine while one annotator with 30% of the queue is systematically wrong.

Worked example

Two annotators, 100 items, a yes/no label. They both say yes on 45, both say no on 30, and they split on the remaining 25 — annotator A alone says yes on 15, annotator B alone says yes on 10.

Raw agreement is $75/100 = 0.75$ , which reads as respectable.

Now the chance correction. A says yes on 60% of items, B on 55%. If they labelled independently at those rates they'd agree by luck on $0.60 \times 0.55 + 0.40 \times 0.45 = 0.51$ of items. So

\kappa = \frac{0.75 - 0.51}{1 - 0.51} = \frac{0.24}{0.49} = 0.49

0.49 is moderate at best, and it means your model's ceiling is somewhere near 75% accuracy. Reporting the 0.75 and shipping is how the four-month project above happened.

Show Me the Code

1import numpy as np2
3both_yes, both_no, a_only, b_only = 45, 30, 15, 104n: int = both_yes + both_no + a_only + b_only5
6def cohens_kappa() -> float:7    """Agreement above what two annotators would reach guessing at their own rates."""8    po = (both_yes + both_no) / n9    a_yes, b_yes = (both_yes + a_only) / n, (both_yes + b_only) / n10    pe = a_yes * b_yes + (1 - a_yes) * (1 - b_yes)11    return float((po - pe) / (1 - pe))12
13print(n, (both_yes + both_no) / n)             # -> 100 0.75    raw agreement looks fine14print(round(cohens_kappa(), 4))                # -> 0.4898      moderate at best15print(round(1 - (both_yes + both_no) / n, 2))  # -> 0.25        a quarter of items disputed16print(np.round([60 / n, 55 / n], 2).tolist())  # -> [0.6, 0.55] the two base rates

Two numbers from the same table. One says the task is fine and one says it isn't.

Watch Out For

Reporting raw agreement on an imbalanced task

"Our annotators agree 96% of the time." On a task where 95% of items are negative, that's what two people who always say negative would achieve.

Kappa exposes it immediately. With a 95% negative base rate, expected chance agreement is about 0.905, so 96% raw agreement gives $\kappa = (0.96 - 0.905)/(1 - 0.905) = 0.58$ — usable, and a long way from what 96% implied. Push the raw figure to 0.955 and kappa drops to 0.53. The headline number is almost entirely the base rate.

The version that costs money is scaling up on the strength of it. You green-light 100,000 labels on a task whose positive class nobody can agree on, and the model that follows is a random number generator on the only class you cared about.

Report kappa alongside raw agreement, always, and report agreement on the positive class separately — that's the number that predicts whether your model can find it.

Averaging away the disagreements

Two annotators split on 25 items, so a script takes the majority, or the first annotator, or a coin flip, and the dataset comes out clean with one label per row.

Two things were destroyed. The information that those items are ambiguous, which was the most useful signal in the whole exercise — those 25 are exactly the examples whose guideline is broken. And the honesty of your evaluation: an ambiguous item in the test set contributes noise to every metric, so a real 3-point improvement can vanish inside it.

Worse, the resolution is often systematic rather than random. "Take annotator A" imports A's bias on every hard case, and A is one person.

Keep the raw per-annotator labels, always, in addition to whatever aggregate you train on. Flag disputed items and exclude them from the test set, or keep them with a soft label. Then route them back into the guidelines, because a disputed item is a specification bug with an example attached.

The Quick Version

A supervised model approximates its labels, so label quality is the ceiling rather than an input.
500 double-labelled examples measure that ceiling for a fraction of what the labelling cost.
The guidelines are the product. Draft, double-label 100, read every disagreement, patch the guidelines, repeat.
Raw agreement flatters imbalanced tasks. Cohen's kappa subtracts chance agreement at each annotator's own base rate.
75% raw agreement with base rates of 0.60 and 0.55 gives a kappa of 0.49, which is moderate at best.
Rough kappa reading: under 0.4 the task isn't defined, 0.6 to 0.8 usable, over 0.8 good.
A hidden gold set gives per-annotator accuracy, which aggregate agreement hides.
Random label noise costs sample efficiency. Systematic noise gets learned faithfully and never averages out.
Keep per-annotator labels and flag disputed items. Don't put ambiguous examples in the test set.