Inter-Annotator Agreement
How often two or more people independently give the same label to the same item — and therefore how well the task is defined.
It's a ceiling, not a diagnostic. A supervised model approximates its labels, so if two annotators disagree on a quarter of the items, no model trained on them can be much more accurate than that.
Raw agreement flatters imbalanced tasks badly — at a 95% negative base rate, two people who always say negative agree 95% of the time. Cohen's kappa fixes that by subtracting the agreement expected from chance at each annotator's own rate: . Rough reading: under 0.4 the task isn't defined, 0.6 to 0.8 is usable, above 0.8 is good. Fleiss' kappa handles more than two annotators, Krippendorff's alpha handles ordinal and missing data.
The point of measuring it is to fix the guidelines. Every disagreement is either an annotator error or an unwritten edge case, and it's usually the second.