AI Benchmarks Explained: What MMLU, SWE-bench and Arena Scores Mean

Short answer: An AI benchmark is a fixed test. A score tells you how a model did on that one test, under the settings the tester chose. It does not tell you how the model will do on your work.

Most launch posts quote the same handful of tests. Each one measures something narrow. Once you know what each test checks, a wall of numbers becomes much easier to read.

This guide covers what the common benchmarks measure, where they break, and a short checklist for using scores when you pick a model. We don't list current scores here, because they go stale within weeks. For current rankings, see our guide to the best AI tools for developers on the Arena leaderboard.

---

Quick Guide to the Common Benchmarks

Benchmark	What it tests	Format	Main weakness
MMLU	General knowledge across 57 subjects	Multiple choice, 4 options	Top models score so high it no longer separates them
MMLU-Pro	Harder knowledge and reasoning	Multiple choice, 10 options	Still multiple choice, still public
GPQA	Expert science questions	448 multiple choice questions (198 in GPQA Diamond)	Small, so a few questions move the score
Humanity's Last Exam	Expert questions across 100+ subjects	2,500 closed-ended questions, some with images	Hard to check every answer key
SWE-bench Verified	Fixing real bugs in Python projects	500 tasks, judged by tests	Signs of contamination and flawed tests
ARC-AGI	Learning a new rule from a few examples	Grid puzzles, and interactive games in version 3	Narrow skill, far from daily tasks
AIME	Competition math	15 questions per exam, integer answers	Few questions, and past exams are online
Arena	Which answer people prefer	Anonymous head-to-head votes	Measures preference, not correctness

---

Knowledge Tests

MMLU: the old standard

MMLU (Massive Multitask Language Understanding) came out in 2020. Its paper describes 57 tasks, from elementary math to US history, computer science and law. Each question has four answer choices.

For years it was the headline number in every model launch. Now the top models all score near the top of the scale, so the gaps between them are small and mostly noise. This is called saturation. A saturated test can still catch a weak model, but it can't tell you which strong model is best.

MMLU-Pro: the harder version

MMLU-Pro was built to fix that. Its authors raised the answer choices from four to ten, removed easy and noisy questions, and added questions that need more reasoning. In their paper, models scored 16% to 33% lower than on MMLU.

The paper also found that scores moved less when the prompt wording changed: about 2% on MMLU-Pro versus 4% to 5% on MMLU. That matters. If a small change in wording moves a score by 5 points, a 3-point lead tells you little.

GPQA: questions experts get right and Google can't answer

GPQA is a set of 448 multiple choice questions in biology, physics and chemistry, written by domain experts. The paper reports that experts with or working toward a PhD in the field got 65% right. Skilled non-experts got only 34%, even after more than 30 minutes per question with full web access.

That is why model makers like it. A high score suggests the model can do more than look things up. Launch posts often quote GPQA Diamond, a smaller, harder subset. GPQA Diamond has only 198 questions, so about two questions move the score by a point.

Humanity's Last Exam: the current hard test

Humanity's Last Exam (HLE) comes from the Center for AI Safety and Scale AI. It has 2,500 questions across more than a hundred subjects, written by nearly 1,000 experts. The questions are closed-ended, so each has one answer that can be checked. Some include images.

HLE also reports calibration error. Models give an answer plus a confidence level, and the test checks whether that confidence matches how often they are right. A model that says "90% sure" and is right half the time is badly calibrated. For everyday use, this matters as much as the raw score, because it tells you how far to trust a confident answer. The team also keeps a private set of questions to check whether models are overfitting to the public ones.

---

Coding Tests

SWE-bench and SWE-bench Verified

SWE-bench is closer to real work than a quiz. Its paper collects 2,294 problems from real GitHub issues in 12 popular Python projects. The model gets the code and the issue, then must edit the code to fix it. A task counts as solved only if the project's tests pass.

In 2024, OpenAI and the SWE-bench authors released SWE-bench Verified, a set of 500 tasks that professional developers checked by hand. It became the main coding number in launch posts.

It has problems now. In February 2026, OpenAI said it would stop reporting SWE-bench Verified. Its team reviewed a set of the hardest tasks and found that many had flawed tests: some rejected correct fixes, and some checked things the issue never asked for. It also found that some models could reproduce the original fixes from memory, a sign that the tasks had leaked into training data. OpenAI now points to SWE-bench Pro, a newer and harder set.

So treat a high SWE-bench Verified score as "this model is good at Python bug fixes in well-known projects," not as "this model can do your engineering work." For coding agents in particular, also read how to evaluate AI agents.

---

Reasoning and Math Tests

ARC-AGI: learning a new rule

ARC-AGI was introduced by François Chollet in 2019. Its idea is "easy for humans, hard for AI." Each task shows a few examples of colored grids being changed by a hidden rule. The model must work out the rule and apply it to a new grid. The tasks avoid language and cultural knowledge, so memorized text helps less. The ARC Prize team keeps a private set of tasks for official scores, so models can't train on the exact test.

There are three versions. ARC-AGI-3 moves from static puzzles to interactive games, where an agent has to learn by trying things inside each environment.

The ARC Prize leaderboard plots score against cost per task. That is a useful habit to copy. A model that gets a few more points by spending far more compute per answer may not be the better choice for you.

AIME: competition math

AIME is a real high school math competition run by the MAA. Each exam has 15 questions, takes 3 hours, and every answer is a whole number from 0 to 999. Because answers are exact, scoring is easy, which is why AI labs use it.

The weaknesses: there are few questions per year, so small differences are noise, and past exams and solutions are all over the internet. A score on a past year's exam may partly reflect memory. Scores on the newest exam, taken after the model was trained, are more meaningful.

Math tests like AIME are where reasoning models shine. If you're not sure what makes those models different, read reasoning models vs regular models.

---

Arena: Scores From Human Votes

Arena (called LMArena until January 2026, and Chatbot Arena before that) works differently from every test above. There is no answer key.

A user types a prompt and sees two anonymous answers side by side. They vote for the better one, and only then are the model names shown. Arena's FAQ says only votes cast while models are anonymous count toward the public rankings. The votes feed a Bradley-Terry rating system, which is similar to the Elo ratings used in chess.

What this tells you: which model people prefer in a blind test, across many kinds of prompts. That is useful, because it is close to how you use a chatbot.

What it doesn't tell you:

• Correctness. People can prefer a confident, well-formatted answer that is wrong.
• Style effects. Longer answers with headings and lists tend to win votes. Arena offers a "style control" view that adjusts for this, and rankings can shift between the two views.
• Small gaps. Two models a few points apart may be tied in practice. Look at the confidence range, not only the rank.
• Your task. Use the category filters (coding, math, writing, and others) instead of the overall board when you have a specific use.

---

Three Problems With Every Benchmark

1. Saturation

When top models all score near 100%, the test stops helping you choose. That happened to MMLU, and it is why new tests like HLE and ARC-AGI-3 keep appearing. A good sign of a still-useful test is that the best models are far from the top of the scale.

2. Contamination

Most benchmarks are public. If test questions, or their answers, end up in a model's training data, the model can score well by memory. The SWE-bench Verified findings above are an example. Private held-out sets (like HLE's) and fresh questions (like a new AIME) help reduce this.

3. Vendor-reported vs independent scores

The numbers in a launch post come from the company selling the model. They choose which tests to show, the prompt format, how many tries the model gets, and how much thinking time it uses. None of that is cheating, but it is not a neutral test either.

Look for independent results from groups that run the same setup on every model. Arena, the ARC Prize leaderboard and the official SWE-bench leaderboards are examples. When the vendor number and the independent number differ, trust the independent one.

---

How to Read a Score When Picking a Model

Use this checklist the next time a launch post shows you a chart.

1. Match the test to your task. Coding? Look at coding tests. Writing and everyday chat? Arena's category boards matter more than GPQA.
2. Check who ran it. Vendor-reported or independent? Prefer independent.
3. Check the settings. Same thinking time, same number of tries, same tools? If one model got more help, the comparison isn't fair.
4. Ignore small gaps. A 1 to 3 point lead on a saturated or small test is usually noise.
5. Look at cost and speed. A slightly higher score at many times the price is often the worse deal.
6. Ask whether the test is fresh. Older, public tests are more likely to be contaminated.
7. Test it yourself. Take 5 to 10 real tasks from your own work, run them on the two or three models you're considering, and compare. This beats any leaderboard for your use.

If you want the background on why models behave this way, our free How LLMs Work course explains it in under an hour.

---

Frequently Asked Questions

What is an AI benchmark?

An AI benchmark is a fixed set of tasks with a clear way to score answers. Labs run their models on it to compare them. Examples include MMLU for general knowledge, SWE-bench for fixing code and Arena for human preference.

Which AI benchmark is the most reliable?

No single benchmark is reliable on its own. Independent tests that keep some questions private, like Humanity's Last Exam and ARC-AGI, are harder to game. Arena is useful for everyday chat because it uses blind human votes. The most reliable test is a small set of your own real tasks.

What does MMLU measure?

MMLU measures general knowledge across 57 subjects, such as math, history, law and computer science, using four-option multiple choice questions. Top models now score so high that MMLU no longer separates them well. MMLU-Pro, with ten options and harder questions, is the more useful version today.

What is SWE-bench Verified?

SWE-bench Verified is a set of 500 real bug-fix tasks from Python projects, checked by professional developers. A model passes a task only if the project's tests pass after its fix. In 2026 OpenAI stopped reporting it, citing flawed tests and signs that tasks had leaked into training data, and pointed to the harder SWE-bench Pro.

How does Arena (LMArena) rank models?

Users see two anonymous answers to the same prompt and vote for the better one. Only votes cast while the models are anonymous count. Arena turns these votes into ratings with a Bradley-Terry model, which is similar to chess Elo ratings. It measures preference, not correctness.

Why do companies report different benchmark scores for the same model?

Scores depend on settings: the prompt format, how many attempts the model gets, how long it can think and which tools it can use. Vendors pick their own settings and often their own choice of tests. Independent leaderboards use one setup for every model, so their numbers are easier to compare.

---

Keep Learning

• How LLMs Work: what is happening inside a large language model, in plain language
• Reasoning Models Explained: why thinking models score higher on math and logic tests

Both courses are free, with a certificate when you finish.

---

Benchmark details checked against each benchmark's own paper, site or announcement in October 2026. Benchmarks and leaderboards change often.