Most of our code is written and reviewed by AI agents, so "which model sits in which chair" is not a trivia question for us. It is staffing.

In July we promoted a shiny new Claude Opus that topped every public leaderboard. A week later we demoted it. It had made confident mistakes in exactly the code where mistakes are expensive: the part of our app that blocks your apps and is not supposed to be talked out of it.

Lesson learned: a leaderboard grades someone else's exam. So we wrote our own.

The exam

Seven bugs we had already caught and fixed, each with a post-mortem that pins down the right answer. Two are "here is an incident, explain it." Five are "here is the pull request just before a reviewer caught the bug: ship or no-ship?" Every one of those five hides a real blocker, so the right answer is always no-ship. The models did not know that. Some of them really did not know that.

Each model got a frozen copy of the repository from just before the fix, read-only tools, no memory, and a 600-word limit. We logged every command and voided any run that peeked outside its copy. Five contestants, 16 runs each: Claude Opus 4.8, Claude Opus 5.5, Claude Fable 5.1, GPT-5.6 Sol and GPT-6 Astra.

Then two graders, one from each family (Astra and Opus 5.5), scored every answer blind, with model names stripped and answers shuffled.

The scoreboard

Found and verdict counts are the GPT grader's; the Claude grader was the same or a little stricter.

Both graders ranked the five in the same order. Yes, the Claude grader put two GPT models above its own relatives, itself included, and we respect the honesty. Slightly less honest: it gave Opus 5.5 about 13 points more than the GPT grader did, while scoring GPT answers within a point and a half. Even judges have family.

Finding it is not the same as saying it

The column that surprised us is "right verdict." The Claude models often found the bug, described it thoughtfully in paragraph four, and then voted to ship anyway. Picture a smoke detector that emails you a nice essay about combustion. The GPT models said no-ship on nearly every real blocker.

The gap lived in the hard cases. On two concurrency races in our iOS background extensions, the Claude models found the bug in 2 of 12 runs combined. The GPT models found one of them every time.

That race was a replay of a real review from this summer: a Claude reviewer walked the code and called it safe, and the Codex reviewer caught it. In the benchmark, history repeated itself. Astra described the exact sequence in both of its runs.

The newest model has a lot to say

Everyone got the same 600-word limit. Four models stayed under it every time, and the GPT models typically used about a third of it. Opus 5.5 went over in 8 of 16 answers, topping out at 800 words.

This is the fastest model in the test (87 seconds a run) and the best bug finder in the Claude family, and it treats a word limit as friendly advice. In a reviewer that is almost charming. In a model writing code from a spec, an instruction treated as a suggestion is the exact failure that got its predecessor demoted in July. We are keeping an eye on it.

And the confident errors, claims stated as certain that turned out to be wrong: Astra made zero under both graders. Opus 4.8 made 20 under the GPT grader.

What we changed

• Our Claude agents moved from Opus 4.8 to Opus 5.5. More bugs found (13 against 10), less than half the cost ($13.84 against $32.62 for 16 runs), and nearly three times faster. We noted the word problem and switched anyway.
• GPT-6 Astra became our routine second reviewer, replacing Sol. Sol found a bit more; Astra made no confident errors and ran three times faster, at roughly the same estimated cost.
• Every pull request keeps two reviewers from different families. This is the best argument for that rule we have: on the hardest bugs, the two families miss different things.
• Any future model swap takes this exam first.

The fine print

Seven cases is a small exam: gaps under about 15 points are noise, though the order held under both graders. These are our bugs (enforcement and concurrency), not a verdict on which model is best in general. And we measured reviewing, not building. A model that spots a race can still write one. A round where each model writes the fix is designed, and it costs far more, so it waits its turn.

Steal this

Pick bugs you already understand. Freeze the repository just before each fix. Make sure the fix is truly out of reach. Give everyone the same prompt. Grade blind, with graders from more than one family. It is the only model leaderboard we trust, mostly because we wrote the answer key.