Inter-Annotator Agreement
How often two or more people independently give the same label to the same item — and therefore how well the task is defined.
Think of It Like This
Like asking three movie critics to independently rate a film: if all three give it two stars, the standard was clear; if they give 1, 3, and 5, the prompt was ambiguous.
It's a ceiling, not a diagnostic. A supervised model approximates its labels, so if two annotators disagree on a quarter of the items, no model trained on them can be much more accurate than that.
Raw agreement flatters imbalanced tasks badly — at a 95% negative base rate, two people who always say negative agree 95% of the time. Cohen's kappa fixes that by subtracting the agreement expected from chance at each annotator's own rate: . Rough reading: under 0.4 the task isn't defined, 0.6 to 0.8 is usable, above 0.8 is good. Fleiss' kappa handles more than two annotators, Krippendorff's alpha handles ordinal and missing data.
The point of measuring it is to fix the guidelines. Every disagreement is either an annotator error or an unwritten edge case, and it's usually the second.