He Wants AI Benchmarking To Be a Science, Not a Dark Art

AI benchmarks are standardized tests designed to evaluate the performance of artificial intelligence models.

They work by splitting data into a training set and a test set.

Models are trained on the training set without restrictions, while the test set is used only once to compare and rank models on a leaderboard.

The first benchmark was created in 1959 to test optical character recognition (OCR) systems on punch cards.

Over time, benchmarks have become the primary way to measure AI progress, driving advancements like the deep learning revolution in 2012.

Moritz Hardt, a computer scientist at the Max Planck Institute for Intelligent Systems, argues that while benchmarks have historically driven progress, they are now fundamentally flawed.

He describes them as 'the mistake that made machine learning' because they prioritize competition over scientific rigor.