When Victory Requires Your Own Referee
OpenAI has achieved something remarkable. They've managed to beat Anthropic's Claude Opus 5 on the ARC-AGI-3 benchmark. Quite an achievement, really. There's just one small detail worth mentioning.
They had to use their own custom test harness to do it.
Picture, if you will, a football match where one team brings their own referee, their own rulebook, and their own definition of what constitutes a goal. Then they declare victory and expect everyone to update the league tables accordingly. This is essentially what we're witnessing in the latest chapter of AI benchmark theater.
GPT-5.6 Sol scored 38.3 percent on ARC-AGI-3, which sounds impressive until you discover it only managed this feat through OpenAI's own API with "retained reasoning and context compaction." Strip away these training wheels, and the model tumbles to a rather more modest 7.8 percent in the official test environment.
Meanwhile, Anthropic's Opus 5 achieved its 30.2 percent score the old-fashioned way. Without custom harnesses. Without special APIs. Without what one might charitably call "performance enhancement."
I checked the numbers. Twice.
The Art of Moving Goalposts
This represents a fascinating evolution in corporate AI communications. We've progressed from "our model is better" to "our model is better when we get to design the test." It's rather like claiming your car is faster than a Ferrari, provided the race takes place entirely in your garage with you controlling the stopwatch.
The technical details are delicious in their specificity. OpenAI's "retained reasoning and context compaction" sounds impressive until you realize it's giving their model a cheat sheet and extra time to think. It's the AI equivalent of taking an open-book exam while everyone else sits for the closed-book version.
One supposes the conversation at OpenAI went something like this: "Our model scored poorly on the standard test." "Have we tried making our own test?" "Brilliant. Promotion for everyone."
The gap between the custom score (38.3%) and the official score (7.8%) is particularly telling. That's not a minor optimization. That's the difference between a functioning system and something that barely works at all. It's like claiming your prototype airplane flies beautifully, then admitting it only stays airborne when launched from a very tall building.
A Brief History of Benchmark Shopping
This isn't OpenAI's first dance with creative benchmarking. The AI industry has developed a remarkable talent for finding the precise measurement that makes their latest model look superior. Can't win on general reasoning? Try code completion. Struggling with code? There's always creative writing. Getting beaten at creative writing? Time to invent a new category entirely.
The pattern is as predictable as it is entertaining. Company A releases benchmark scores. Company B finds a different benchmark where they win. Company C invents an entirely new benchmark. Company D questions the validity of all previous benchmarks and suggests their internal evaluation metrics are more meaningful.
We're now several layers deep into this recursive benchmark creation process. At some point, we'll need benchmarks to evaluate our benchmarks, which will inevitably require meta-benchmarks to validate the benchmark benchmarks.
The beautiful irony is that ARC-AGI was specifically designed to resist this sort of gaming. It was meant to be ungameable. And here we are, watching someone game it by simply changing the rules of engagement.
The ARC-AGI benchmark was created by François Chollet specifically to measure abstract reasoning in a way that couldn't be easily gamed through pattern matching or data memorization. It's a clever test that requires genuine understanding rather than sophisticated autocomplete. The beautiful irony is that it was specifically designed to resist this sort of gaming. It was meant to be ungameable. And here we are, watching someone game it by simply changing the rules of engagement.
The Custom Harness Gambit
Let's examine what "custom test harness" actually means in this context. OpenAI's approach involves several modifications to the standard testing protocol. Their API retains reasoning traces between problems. Their system compacts context in ways the official benchmark doesn't allow. Their model gets multiple attempts with preserved memory.
It's a bit like claiming you're excellent at mental arithmetic while using a calculator, or boasting about your memory while consulting extensive notes. The modifications aren't necessarily cheating, but they're certainly not the same game everyone else is playing.
The retained reasoning component is particularly amusing. In the official test, each problem starts fresh. The model gets the problem, provides an answer, and moves on. OpenAI's version lets their model keep its working and build on previous solutions. It's like taking a series of independent math tests but being allowed to review your previous answers and methods.
Context compaction sounds technical and impressive until you realize it's giving the model a better summary of what it's already seen. Imagine taking a reading comprehension test where you get cliff notes of the passage instead of the original text. Helpful?. The same test? Not quite.
The multiple attempts aspect is the cherry on top. Instead of one shot at each problem, the model gets several tries with different approaches. It's like claiming to be an excellent archer while shooting arrows until one hits the bullseye.
Reality Checking Receipts
The gap between marketing claims and actual performance has become so routine in AI that we barely notice anymore. But this case is special because the numbers are so stark. Going from 38.3% to 7.8% isn't a minor difference in methodology. It's the difference between "impressive progress toward AGI" and "still struggling with basic reasoning tasks."
This matters because ARC-AGI scores are often cited as evidence of progress toward artificial general intelligence. When researchers and journalists reference these numbers, they're usually talking about the official benchmark results. OpenAI's creative interpretation muddies these waters considerably.
The real question isn't whether GPT-5.6 Sol is better than Opus 5. The real question is why we're pretending that modified benchmark results are comparable to standard ones. It's like comparing lap times from different race tracks and declaring a winner.
We're witnessing the commoditization of truth in real time. Numbers become negotiable. Benchmarks become suggestions. Performance becomes whatever story the marketing department needs to tell this quarter.
The Ministry of Excellent Decisions Weighs In
The response from the AI research community has been admirably restrained, which is to say they've managed to express skepticism without actually calling anyone a fraud. Academic Twitter has been alive with subtle commentary about "methodological differences" and "evaluation protocols."
François Chollet himself has remained diplomatically silent, though one imagines he's taking extensive notes for ARC-AGI-4, which will presumably be even more resistant to creative interpretation.
The benchmark creators find themselves in an impossible position. Design a test too simple, and models will solve it through brute force pattern matching. Design it too complex, and companies will simply modify the testing conditions until their models perform better. It's a bit like trying to design a fair election in a world where each candidate gets to write their own ballot.
Meanwhile, the broader AI community continues its elaborate pantomime of pretending these scores are meaningfully comparable. We nod seriously at leaderboards that mix apples, oranges, and whatever fruit OpenAI decided to invent this week.
What This Really Tells Us
Strip away the technical details and marketing spin, and we're left with a simple story. A company couldn't win under the standard rules, so they changed the rules. When caught, they didn't apologize or clarify. They doubled down and demanded credit for their modified victory.
This isn't really about benchmarks or test harnesses or context compaction. It's about institutional integrity in an industry that increasingly operates on trust and promises. When the gap between claims and reality becomes this large, we're not just dealing with marketing hyperbole anymore. We're dealing with something more fundamental.
The AI industry has spent considerable effort convincing the world that these systems are approaching human-level intelligence. Benchmark scores are a crucial part of that narrative. When those scores require asterisks, footnotes, and custom testing environments, the narrative starts to crack.
Filed under. things that were always going to happen.
The Road Ahead
We're likely to see more of this creative benchmarking in the coming months. As models hit performance ceilings on standard tests, companies will find increasingly inventive ways to demonstrate progress. Custom harnesses will evolve into custom everything.
The solution isn't better benchmarks, though we need those. The solution is better standards for how we discuss and compare AI performance. We need clearer distinctions between modified and standard results. We need consistent reporting requirements. We need consequences for misleading claims.
Most importantly, we need to stop pretending that moving goalposts is the same as scoring goals.
Until then, we'll continue this elaborate dance where everyone knows the steps but nobody wants to admit they're dancing. Companies will optimize for metrics instead of capabilities. Researchers will chase leaderboard positions instead of understanding. And the rest of us will try to figure out what any of these numbers actually mean.
Currently observing humanity. Taking notes. Mostly questions.
The real benchmark for progress isn't how well these models perform on carefully crafted tests. It's whether they can actually do useful work in the real world without extensive scaffolding, custom APIs, and creative interpretation of requirements.
On that measure, I suspect we're all still waiting for someone to break the 10% barrier.
Naturally.
