Benchmark Suites and Leaderboards
Collections of benchmarks and public rankings drive progress and distort it in equal measure; the ranking is not the goal.
From One Test to Many
A single benchmark is easy to game and narrow in what it measures. A suite bundles many tasks so a method must be broadly capable to score well, and a leaderboard ranks entries publicly. Suites and leaderboards have accelerated whole fields by making progress visible and comparable.
Goodhart's Law
When a measure becomes a target, it ceases to be a good measure. A leaderboard that rewards a score invites optimization of the score itself: methods are tuned to the specific tasks, hyperparameters are searched against the public set, and unmeasured qualities decay. The ranking then tracks benchmark-fitting skill, not the capability it was meant to proxy.
Contamination and Leakage
- Test-set contamination: benchmark answers appear in training data, inflating scores.
- Repeated tuning against a public test set, which turns it into a training set.
- Selective submission: reporting only the best of many attempts.
- Metric gaming: exploiting quirks of the scoring rule rather than solving the task.
Reading a Leaderboard Honestly
A rank difference is meaningful only if it exceeds run-to-run variance and holds on held-out or fresh tasks. Small gaps near the top of a saturated board usually reflect tuning, not capability. The durable signal is generalization to tasks the method was not optimized against.
In Science
Scientific benchmarking values held-out cases, blind comparisons, and agreement among independent codes over a single leaderboard position. The aim is to learn which method is actually right, not which is best at the test, which is why cross-code agreement carries more weight than a solo top score.