Benchmarks and what they fail to measure

Every model release comes with a table of benchmark scores. Reading those tables usefully requires knowing what each number can and cannot support.

Contamination

The most serious methodological problem in the field.

Models are trained on very large web scrapes. Public benchmarks are on the web. Which means benchmark questions and answers can end up in training data.

A model that has memorised a test set scores brilliantly and has demonstrated nothing.

Detection is difficult. Techniques include checking for verbatim reproduction, comparing performance on original versus paraphrased items, and testing on data created after the training cutoff.

These help and none is conclusive, because contamination can be partial — the model may have seen discussion of the questions rather than the answer key, which still inflates the score.

The practical response is to weight benchmarks created or refreshed after a model's training cutoff far more heavily than long-established ones.

Saturation

A benchmark stops being informative once the leading models score near the ceiling.

Several once-standard tests are now effectively saturated, with the top of the leaderboard separated by margins smaller than the label error rate in the dataset itself.

At that point the remaining differences measure noise and annotation quality rather than capability.

This is why benchmark suites turn over so quickly, and why a model advertised as leading on an older test may simply be leading on a test that stopped discriminating.

Construct validity

The question of whether the test measures the thing it is named after.

A multiple-choice examination of factual knowledge measures recall of that domain's facts. Whether it measures reasoning depends entirely on the items, and multiple-choice format allows strategies — eliminating implausible options — that do not require solving the problem.

Similarly, a coding benchmark composed of self-contained functions with unit tests measures something real and quite different from working in a large existing codebase.

The gap between benchmark performance and practical usefulness usually lives here.

The evaluation conditions

Numbers are not comparable unless the conditions are.

Prompting. The same model scores differently with different prompts. Chain-of-thought prompting, few-shot examples and formatting all move results substantially.

Sampling. Temperature and sampling strategy affect results, and reporting the best of several attempts is a different measurement from reporting a single attempt.

Tool access. A model with a calculator and a search tool is a different system from the model alone.

Compute at inference. Approaches that spend more computation per query — sampling many candidates and selecting among them — trade cost for accuracy. Comparing such a system to a single-pass model on accuracy alone omits the cost axis entirely.

Good evaluations state all of this. Marketing tables frequently do not.

What is more informative

Held-out and refreshed benchmarks whose items are not public, or are regenerated periodically.

Human preference comparisons at scale, where anonymised model outputs are rated head to head. These capture qualities that automatic metrics miss and introduce their own biases toward length and confident tone.

Task-specific evaluation on your own data. The only measurement that reliably predicts whether a model will work for your purpose.

A few hundred representative examples from your actual workload, scored by someone who knows the domain, is worth more than any published leaderboard.

Failure analysis rather than aggregate scores. Where a model fails is more informative than how often, because it tells you whether failures are tolerable.

Leaderboards and their incentives

Worth noticing as a structural matter.

Public leaderboards create an incentive to optimise for the leaderboard, which is a well-documented dynamic in every field that has adopted them.

The effect is not usually deliberate cheating. It is a gradual drift of research effort toward what the benchmark measures and away from what it does not, and a tendency to report the evaluation on which a model happens to do well.

Which is an argument for evaluating on a suite rather than a single number, for refreshing benchmarks regularly, and for treating a large lead on one test with more suspicion than a consistent showing across several.

The reporting norms worth demanding

Which benchmarks, at which version, run when.

Prompting method, stated.

Whether tools were available.

Compute or cost per query, so accuracy and expense can be traded explicitly.

Confidence intervals, since many reported differences fall within sampling noise.

Contamination checks.

Papers from serious groups increasingly report these. Product announcements generally do not, and the absence is itself informative.