Iro AIIro AI Blog
Every model launch arrives with a bar chart, and the bar chart is a marketing asset before it is a measurement. You do not need to understand the benchmarks to stop being misled by them. You need about five questions.
A benchmark score is not a property of a model. It is the result of one specific model, run through one specific piece of software, on one specific day, scored one specific way. Change any of those and the number moves. Launch charts rarely tell you which ones changed.
You do not need to know what SWE-bench or ARC-AGI measures to read these charts usefully. You need to know the handful of places the number can be quietly improved before it reaches you, and to ask whether they were.
The single most useful habit: when a chart compares one company's model against its rivals, check whether every bar was produced under the same conditions. Often they were not, and that is usually where the lead comes from.
A harness is the software wrapped around a model during a test. It decides which tools the model can reach, what it remembers between steps, and how a long conversation gets trimmed when it runs out of room. None of that is the model. All of it changes the score.
September 2026 produced a clean illustration. OpenAI's GPT-6 Astra was reported at 99.9% on ARC-AGI-3, against 30.2% for Claude Opus 5 and 7.8% for GPT-5.6 Sol. That reads as a benchmark being retired.
ARC Prize, who run it, published the fuller picture. The 99.9% came from OpenAI's own provider adapter, which preserves the model's internal reasoning state between actions and compacts long conversations for it. Run through ARC Prize's standard harness, which gives every lab's model the same minimal interface, Astra scored 62.7%.
Sit with how large that gap is. Same model, same benchmark, same week. The difference is entirely in what the surrounding software was allowed to do for it, and it is worth 37 percentage points.
This is not an accusation. Both numbers were published, and 62.7% against 30.2% is still a commanding lead. The failure is in the chart, not the lab: putting a provider-adapter number next to standard-harness numbers produces a comparison that the underlying measurements do not support. ARC Prize's position is that the standard harness is the one that compares across labs, which is why the lower number is the meaningful one here. Figures as reported in September 2026.
Once you know harnesses exist, a lot of launch charts start reading differently. "Our model scored X" and "our model scored X under the same conditions as theirs" are very different claims, and only one of them is a comparison.
Practice this, don't just read it.
Iro AI turns ideas like the ones in this post into 5-minute exercises with feedback. Free tier, Pro from $4.17 a month ($49.99 a year, 7-day free trial).
Outright fabrication is rare. It is also unnecessary, because there are easier moves.
The new model is charted against a competitor's release from eight months ago, because the competitor's current one would narrow the gap. Nothing is false. The chart is simply answering a question nobody asked. Check the dates and version numbers on every bar, not just the winning one.
Some scores let a model attempt a problem repeatedly and count success if any attempt works. That is a legitimate way to measure, and it is a very different thing from what you experience, which is one attempt at a time. Wording like pass@k, best-of-n, majority vote or consensus signals this. A headline number without that wording is not automatically single-attempt; sometimes the wording is in a footnote.
Benchmarks are published on the internet. Models are trained on the internet. When a test's questions and answers end up in training data, the model can score well partly through recall rather than reasoning. Labs work to prevent it and it still happens. It is the main reason a model can look excellent on a famous benchmark and ordinary on your actual work. Newer and private benchmarks are harder to contaminate, which is part of why they exist.
When a benchmark saturates, attention shifts to a new one, and the new one is often chosen partly because the model in question does well on it. Watch for a launch that leads with a benchmark you have never heard of while being quiet about the established ones. That is not disqualifying, but it is information.
In rough order of how much they tell you:
That last one deserves weight. Most benchmark arguments are irrelevant to most users. If you use AI to draft, summarise and think out loud, the differences between frontier models on agentic coding tests will not touch your week. The model you have is almost certainly not your bottleneck.
It is tempting to file benchmark scepticism under industry trivia. It is not. It is the same skill as the rest of using AI well, pointed at a different target.
Reading a benchmark chart critically is reading a confident, well-presented, technically accurate claim and asking what it leaves out. That is precisely what you do with a model's own output: it arrives fluent and certain, and your job is to work out what it did not tell you. People who are good at one tend to be good at the other, because the underlying move is the same one.
It also protects you from the expensive version of the mistake. Switching tools, paying for a higher tier, or rebuilding a workflow on the strength of a bar chart costs real time and money. Catching a confident wrong answer and catching a confident misleading chart are the same reflex.
The useful stance is neither credulity nor cynicism. Benchmarks do measure something real, and the field genuinely moves. They are just evidence that arrives pre-packaged by someone with an interest in your conclusion, and evidence like that gets read carefully.
Practice this, don't just read it.
Iro AI turns ideas like the ones in this post into 5-minute exercises with feedback. Free tier, Pro from $4.17 a month ($49.99 a year, 7-day free trial).
A harness is the software wrapped around a model during a test. It controls which tools the model can use, what it remembers between steps, and how long conversations are trimmed. The same model can score very differently under different harnesses, because the harness is doing part of the work.
The 99.9% came from OpenAI's own provider adapter, which preserves the model's reasoning state between actions. The 62.7% came from ARC Prize's standard provider-neutral harness, the one used to compare models across labs. Both are honest measurements of different conditions.
They are reliable as measurements of the specific thing they test under specific conditions. They become unreliable when used as general claims about which model is better, because conditions vary, comparisons can be stale, and test questions can leak into training data.
Benchmark contamination is when a test's questions and answers end up in a model's training data, so the model can score well through recall rather than reasoning. It is a main reason a model can look excellent on a famous benchmark and ordinary on your real work.
Only as a rough filter. Ask whether the benchmark resembles your actual work, then test the shortlist on two or three real tasks of your own. For everyday drafting and summarising, differences between frontier models rarely change the outcome.