Skip to content
AI360XpertBeta

Measuring General Capability

A high exam score can mean the model memorized the questions. You measure general capability by testing it on tasks it has never seen and watching the gap.

Same model, leaked questions score 0.85 but novel tasks score 0.55, so the 0.30 gap is memorization, not reasoning
Same model, leaked questions score 0.85 but novel tasks score 0.55, so the 0.30 gap is memorization, not reasoning

Why Does This Exist?

Most leaderboards still rank models with fixed public question sets. The trouble is that those questions live on the open internet, and pretraining eats the open internet. A model can answer from memory and look like it reasoned.

This page is about the honest version of the test: fresh tasks the model could not have memorized, a check that the test itself stayed clean, and a way to grade open answers. The leak itself gets a full treatment in benchmark contamination. The grading trick is LLM-as-a-judge. Routine checks after a tune belong to post-training evaluation.

One idea first. Accuracy is the fraction of answers judged right: the correct count divided by the total count. And a score only means what you hope it means when the model never saw the items during training. Everything below is machinery for making that condition true, and for checking that it held.

Think of It Like This

A driving test on your own street

Picture your driving test taking place entirely on your own street. You have driven it a thousand times. You know every pothole, every blind corner, every badly timed light. You pass with ease.

Now the examiner asks you to drive across an unfamiliar town in the rain. Different junctions, surprise roadworks, a detour sign you have never seen. That second drive is the real test of whether you can drive. The first one mostly proved you know your street.

A public benchmark is your street. A fresh task set is the unfamiliar town. Where the analogy stops: a driving examiner watches how you handle surprises, while a benchmark only sees final answers, so a lucky guess and real reasoning score the same unless the task design rules guessing out.

How It Actually Works

Capability testing has three moving parts. You write tasks the model cannot have memorized, you confirm the test stayed clean, and you grade answers that have no single right wording. Miss any one of the three and the number lies.

Score each set the same way

Fix two item sets of matched size and difficulty: one public set the model may have seen, one novel set it cannot have seen. Grade both with the same rubric and write A=C/NA = C / N for each, where AA is accuracy as a fraction, CC is the count judged correct, and NN is the total count. Same size, same difficulty, same grader. If any of those three differ, you are comparing the tests, not the model.

Read the gap

Call the public score ApublicA_{\text{public}} and the novel score AnovelA_{\text{novel}}. The contamination gap is G=Apublic−AnovelG = A_{\text{public}} - A_{\text{novel}}. A gap near zero says the public score transfers. A large positive gap says memory did the work: the model knew the questions, not the skill. That gap is the whole diagnosis, and it is why you never report the public number alone.

Worked example

Take 20 published questions and 20 fresh ones, matched on difficulty and graded by the same rubric. The model gets 17 of the public ones right and 11 of the novel ones right.

Apublic=17/20=0.85A_{\text{public}} = 17 / 20 = 0.85. Anovel=11/20=0.55A_{\text{novel}} = 11 / 20 = 0.55. So G=0.85−0.55=0.30G = 0.85 - 0.55 = 0.30. Thirty points of the public score evaporate on fresh material.

Is that gap just luck? For a fraction near 0.55 over 20 items, the sampling wobble (one standard error) is about 0.55×0.45/20≈0.11\sqrt{0.55 \times 0.45 / 20} \approx 0.11. The observed 0.30 is nearly three times that, so luck is a poor explanation. The honest headline is the novel score, 0.55, not the public 0.85. Those are the numbers drawn in the diagram above.

The trade is cost for trust. Fresh tasks need human authoring and careful grading, often with a judge model checking every answer, so each honest point costs far more than a reused static point. Skip the bespoke set for routine regression checks, where a locked private set that no training run touches does the job. The whole method rests on one assumption: the novel items are truly unseen and equally hard. Break it, by letting harder items into the novel set or leaking them into training, and the gap blames memory for what was really difficulty, or hides the leak entirely.

Code

The snippet below recomputes the worked example, so you can reuse the pattern on your own two sets.

def accuracy(correct: int, total: int) -> float:    """Fraction of items judged right."""    return correct / total

public_score = accuracy(17, 20)novel_score = accuracy(11, 20)gap = round(public_score - novel_score, 2)print(public_score)  # -> 0.85print(novel_score)  # -> 0.55print(gap)  # -> 0.3

The prints read 0.85, 0.55 and 0.3, matching the hand computation above. Point this function at your own matched sets and the gap tells you how much of your public score was recall.

Watch Out For

Training on the test, then celebrating the score

Symptom: your public score climbs week after week while users keep reporting the same failures. Nothing you ship feels smarter, yet the chart says otherwise.

The cause is pressure on the number itself. Once a score decides releases, training quietly absorbs the test: vendor data mixes, synthetic rephrases of test items, just a few examples added to the tune. None of it looks like cheating up close. Together it turns the benchmark into a memory check.

Fix it with procedure, not willpower. Keep one private set that no training run ever sees, rotate fresh items in on a schedule, and treat any score on data that touched training as a debugging signal, never a headline. How leaks happen and how to detect them is the subject of benchmark contamination.

Quoting one average as if ability were flat

Symptom: a model aces exam-style questions, so you trust it with a simple multi-step task, and it collapses on step two. The average promised a generalist. You got a specialist with good recall.

Capability is spiky. Strong recall in one family says nothing about planning or tool use in another, and whole-task success can switch on suddenly with scale while averages move smoothly, which is the subject of emergent abilities.

Fix it by refusing the single number. Report scores per task family plus the novel-task gap, and let the weakest family, not the average, decide what you trust the model to do.

The Quick Version

  • A public benchmark score mixes memory with reasoning whenever the questions could have reached training data.
  • General capability is what survives on fresh tasks: matched size, matched difficulty, same grader, unseen items.
  • The contamination gap is public accuracy minus novel accuracy. Here 0.85 minus 0.55 leaves 0.30 that was recall, not skill.
  • Open answers need a written rubric and a judge, not string match, because correct wording varies.
  • Report per-family scores plus the gap, since one average hides spiky ability and sudden jumps with scale.