Human Evaluation Protocols

Why Does This Exist?

With the rise of automated techniques like llm-as-a-judge, you might assume that human evaluation is obsolete. The exact opposite is true. Automated judges must be calibrated against human ground truth to prove they actually work. If you do not have a robust human evaluation dataset, you cannot trust your automated pipeline.

The problem is that humans are terrible measuring instruments. We are highly subjective, easily distracted, and our definition of "good" drifts throughout the day depending on how tired we are. If you give a thousand model outputs to a junior engineer and tell them to "rate these from 1 to 5," you are not collecting data. You are collecting noise. If you give the same thousand outputs to another engineer, their scores will look completely different.

Human evaluation protocols exist to turn subjective, noisy human opinions into objective, statistically reliable data. By enforcing strict rules around rubric design, overlap, agreement tracking, and adjudication, these protocols squeeze the human error out of the measurement process.

Think of It Like This

How It Actually Works

Running a scientifically valid human evaluation requires moving through four strict phases. If you skip any of these phases, your resulting data is invalid.

1. Rubric Design and Training

You never ask an annotator to rate "quality." You ask them to rate specific, objective criteria. A rubric breaks a subjective task down into mechanical instructions. Instead of "Is this answer helpful?", the rubric asks:

• "Does the answer directly address the user's prompt? (Yes/No)"
• "Does the answer contain hallucinated facts? (Yes/No)"
• "Is the formatting compliant with the requested schema? (Yes/No)"

Before the actual evaluation begins, you write the rubric and give 50 sample examples to your annotators. You then review where they struggled, rewrite the confusing parts of the rubric, and repeat until everyone understands the rules.

2. Overlap and Independence

In a naive setup, if you have 1,000 examples and two annotators, you give 500 to Annotator A and 500 to Annotator B to save money. Never do this.

In a rigorous protocol, you must build in overlap. For example, 20% of the dataset is given to both Annotator A and Annotator B. They must score these overlapping examples completely independently, without seeing each other's answers. This overlap is the only way to measure if your rubric is actually working.

3. Measuring Inter-Rater Agreement

Once the overlap data is scored, you measure how often the annotators agreed. The industry standard metric for this is Cohen’s Kappa (for two raters) or Fleiss’ Kappa (for more than two raters).

Unlike a simple percentage agreement (which can be inflated by chance if the dataset is skewed toward one answer), Kappa mathematically accounts for the probability of the annotators agreeing by random guessing.

• Kappa < 0.2: Slight agreement (Your rubric is broken).
• Kappa 0.4 - 0.6: Moderate agreement (Needs improvement).
• Kappa > 0.8: Almost perfect agreement (You can trust the data).

4. Adjudication

When Annotator A and Annotator B disagree on an overlapping example, the system flags it. A more senior evaluator (the Adjudicator) reviews the example, reads the differing scores, and makes the final, authoritative decision. The adjudicator also provides feedback to the annotators to realign them with the rubric.

Show Me the Code

Here is how you calculate simple percentage agreement and Cohen's Kappa in Python to determine if your human evaluators are producing reliable data.

1from sklearn.metrics import cohen_kappa_score2
3def calculate_annotator_reliability(4    annotator_a: list[int], 5    annotator_b: list[int]6) -> dict[str, float]:7    """8    Calculates agreement metrics for a set of overlapping annotations.9    """10    if len(annotator_a) != len(annotator_b):11        raise ValueError("Annotator lists must be the same length.")12        13    # Calculate simple agreement14    matches = sum(1 for a, b in zip(annotator_a, annotator_b) if a == b)15    simple_agreement = matches / len(annotator_a)16    17    # Calculate Cohen's Kappa (accounts for chance)18    kappa = cohen_kappa_score(annotator_a, annotator_b)19    20    return {21        "simple_agreement": simple_agreement,22        "cohens_kappa": kappa23    }24
25# annotator_a = [1, 5, 3, 2, 5, 1, 4]26# annotator_b = [1, 4, 3, 2, 5, 1, 2]27# -> calculate_annotator_reliability(annotator_a, annotator_b)28# {'simple_agreement': 0.71, 'cohens_kappa': 0.62} 29# (Moderate agreement; the rubric needs work).

Watch Out For

The Quick Version

• Human evaluation is required to ground automated metrics in reality, but human opinions are inherently noisy and subjective.
• A human evaluation protocol is a strict methodology for converting subjective human judgments into objective, reliable data.
• Never evaluate on "vibes"; break the task down into an objective, mechanical rubric.
• Always assign a percentage of the dataset to multiple independent annotators to create overlap.
• Measure the quality of your rubric and your annotators using Cohen's Kappa, which accounts for random chance.
• When annotators disagree on an overlapping example, a senior adjudicator makes the final call.

What to Read Next

• llm-as-a-judge — Once you have a high-Kappa human dataset, you use it to validate whether your automated LLM judge is scoring things correctly.
• evaluation-significance — Why you need enough human annotations to ensure your final metric differences are statistically valid.