He Wants AI Benchmarking To Be a Science, Not a Dark Art

AI benchmarks are standardized tests designed to evaluate the performance of artificial intelligence models. They work by splitting data into a training set and a test set. Models are trained on the training set without restrictions, while the test set is used only once to compare and rank models on a leaderboard. The first benchmark was created in 1959 to test optical character recognition (OCR) systems on punch cards. Over time, benchmarks have become the primary way to measure AI progress, driving advancements like the deep learning revolution in 2012. Moritz Hardt, a computer scientist at the Max Planck Institute for Intelligent Systems, argues that while benchmarks have historically driven progress, they are now fundamentally flawed. He describes them as 'the mistake that made machine learning' because they prioritize competition over scientific rigor.

Benchmarks have played a pivotal role in AI’s history. In 1987, a benchmark for speech-to-text translation showed that AI models performed worse than simple statistical methods, contributing to a 25-year AI winter where funding and interest declined. The 2012 ImageNet benchmark, which tested image classification, sparked a deep learning revolution when an eight-layer neural network outperformed other algorithms. Today, benchmarks for large language models (LLMs) like OpenAI’s GPT-6 Astra or Anthropic’s Claude Fable measure abilities such as solving math problems or finding software bugs. However, many experts now argue that benchmarks have become fatally flawed, with Moritz Hardt stating the benchmarking ecosystem is broken. Despite this, benchmarks have historically driven progress, albeit often by luck rather than design.

Benchmarks are supposed to act like thermometers, measuring how well an AI model performs. However, in practice, they function more like sports competitions or markets, where the rules define the game and participants compete to win. A major issue is that test sets are often reused and made publicly available, allowing model developers to incorporate them into their training loops. This violates the holdout method, a fundamental principle in machine learning where the test set should remain completely separate until final evaluation. As a result, a model’s accuracy can vary dramatically depending on the test set used—for example, 90% on one dataset and 73% on another. This inconsistency makes absolute scores misleading, though relative rankings between models tend to remain stable across different datasets.

While absolute scores on benchmarks are unreliable, rankings—comparisons between models—tend to be stable. If Model A scores 90% on a benchmark and Model B scores 80%, Model A will likely outperform Model B on other datasets as well. This stability has historically helped machine learning avoid a replication crisis, where results cannot be reproduced. Moritz Hardt notes that this reliability is largely empirical and not well understood theoretically. For decades, benchmarks provided a way to identify improvements, even if the underlying mechanisms remained unclear. However, this accidental reliability is now at risk in the era of LLMs, where models are trained on unknown datasets and can be strategically optimized for specific benchmarks.

Large language models (LLMs) have introduced a new challenge to benchmarking. Unlike earlier models, LLMs are trained on vast and often undisclosed datasets, making it impossible to ensure they haven’t indirectly learned from benchmark test sets. This creates an uneven playing field, as some models can be post-trained or benchmaxxed—optimized specifically for certain benchmarks. For example, newer versions of models like Gemini or GPT often show high performance on benchmarks released before their training but lower performance on newer benchmarks. Moritz Hardt compares this to altitude training in sports, where some competitors gain an unfair advantage. The result is that benchmark scores no longer reliably indicate true model capabilities, leading to what Hardt calls a benchmarking crisis.

Moritz Hardt proposes a solution inspired by the 1968 Olympics, where altitude training was allowed to level the playing field. For AI benchmarks, this would mean implementing a standardized preparation protocol—or altitude training—where all models are trained for a short period on benchmark-relevant data before evaluation. This would reduce disparities caused by undisclosed training data and ensure a fairer comparison. However, Hardt acknowledges that even with this approach, benchmarks would still only provide relative rankings rather than absolute measurements. While this restores some validity to comparisons, it does not address the deeper issue of understanding how models work. Benchmarks alone cannot reveal the mechanisms behind AI performance, though they may indirectly contribute to scientific knowledge over time.