One Number Is Not A Result

Why Does This Exist?

Every week, a new model drops with a press release claiming it beat the state-of-the-art on MMLU or HumanEval by 0.5%. The chart shows a bar slightly taller than the competitor's bar. The community cheers, the model trends, and the cycle repeats.

But these single-run benchmark numbers are statistically meaningless. Evaluating a generative model isn't like running a unit test. Due to sampling temperature, prompt sensitivity, and floating-point non-determinism across different GPU clusters, a model's score can swing by several percentage points between runs.

Think of It Like This

Judging an athlete's career based on a single throw in practice is absurd. You look at their average performance and their consistency across a season. But we routinely judge billion-dollar models on a single, cherry-picked "throw" on a benchmark dataset.

How It Actually Works

When a provider claims a score of 91.2%, they are often reporting the max score across hundreds of internal runs, using heavily optimized prompts tailored specifically to their architecture.

A rigorous result requires a spread: the mean score and the standard deviation across multiple seeds, temperatures, and prompt variations. Model A might score 91% with zero variance—it is robust and predictable. Model B might score 91% once, but average 85% with massive variance depending on the phrasing of the question. A single-run table obscures the fact that Model A is vastly superior in production.

Watch Out For

Never trust a benchmark figure without the harness, the settings, and the date. If a provider doesn't publish the exact evaluation code (like their fork of lm-evaluation-harness) and the exact hyperparameter configurations used to achieve the score, the number is marketing, not science.

(Correct as of August 2026).

The Quick Version

A single top-line benchmark score is useless due to natural variance in generative models. True model capability can only be understood by looking at the mean and spread across multiple runs and prompt variations.

What to Read Next

Review the benchmark-contamination and benchmark-saturation pages to see how these evaluations are further compromised.