Benchmark
A standard test used to measure and compare how well AI models perform on a particular task. Benchmarks turn vague claims of capability into numbers — though a high score does not always mean real-world usefulness.
How do you tell whether one AI model is better than another? A benchmark is the usual answer: a fixed set of questions or tasks, with known correct results, that every model can be run against. Because the test is the same for all of them, the scores can be compared, and the field uses benchmarks constantly to track progress and to back up claims that a new model is state of the art.
Benchmarks cover all sorts of skills — solving maths problems, writing code, answering exam questions, reasoning over long documents. Testing a model this way is often called evaluation, or “evals” for short, and running good evals has become a serious discipline in its own right.
They come with a well-known catch, though. A model can be tuned to shine on popular benchmarks without becoming genuinely more capable, and a test can leak into training data, inflating the score. A benchmark result is a useful signal, not a guarantee; the real proof is whether a model actually helps with the task you care about.