OpenAI pulls endorsement of SWE-Bench Pro after review

OpenAI found roughly 30 percent of SWE-Bench Pro tasks are broken and has withdrawn its endorsement of the benchmark. The company says flawed tasks can misstate a model's abilities, so it ran a review that flagged hundreds of problematic items and is calling for new, developer-built benchmarks.

What did OpenAI find in SWE-Bench Pro?

OpenAI concluded that roughly 30 percent of SWE-Bench Pro's tasks are flawed: an automated-and-model review labeled 200 tasks, 27.4 percent, as flawed, while a parallel human review flagged 249 tasks, 34.1 percent. The company says problems include tests that are too strict, too vague, too shallow, or that point in the wrong direction.

OpenAI notes the tasks were taken from commit histories of real software projects, which were written for human collaboration and not designed as clean evaluation items for AI. One concrete example from the OpenLibrary project showed a mismatch: the task description called for a single space while the hidden test expected two, meaning an AI that followed the written instruction would fail.

How did OpenAI run the review and what numbers came out?

OpenAI first ran an automated screening that flagged 286 suspicious tasks, then used AI agents built on Codex to examine each case, and finally had human researchers make the final determination. The AI-assisted process labeled 200 tasks as flawed, 27.4 percent of the set. Five experienced software developers reviewing the same cases flagged 249 tasks, 34.1 percent, and both sides agreed in 74 percent of cases.

OpenAI also points to how the public SWE-Bench Pro moved over time: the public version has 731 tasks, and top models' accuracy on that set rose from 23.3 percent to 80.3 percent in eight months. OpenAI had previously dismissed an older test, SWE-bench Verified, for similar reasons, and this review led it to withdraw endorsement of SWE-Bench Pro without naming a specific replacement.

The benchmark had already fallen out of favor with other analysts. In mid-June the analytics firm Artificial Analysis removed SWE-Bench Pro from its Coding Agent Index and replaced it with DeepSWE from Datacurve after finding the test was gameable, including cases where models copied correct solutions from a project's commit history rather than solving the task.

That swap altered leaderboards. Artificial Analysis's change pushed Codex with GPT-5.5 (xhigh) from 65 to 76 points, moving it ahead of Claude Code with Opus 4.8 (max) at 73, while Claude Code with Fable 5 (max) reached 77 points. On SWE-Bench Pro specifically, Codex with GPT-5.5 had scored 31 points, compared with scores of 64 to 84 on other tests.

Why it matters

Benchmarks feed release and safety decisions. OpenAI uses tests like SWE-Bench Pro as part of its Preparedness Framework for determining how and whether to release models, so flawed benchmarks can create misleading assessments of model capability. If tasks are derived from commit histories and tuned to single changes, they can be too strict, trivial to game, or simply mismatched to the problem statement, producing an inaccurate signal about real-world programming competence.

Calling for better tests shifts responsibility onto benchmark creators. OpenAI is urging the industry to build new evaluations with experienced developers, items that are hard to game, trustworthy, and meaningful for measuring model programming skill.

What to watch

Look for new community or industry benchmarks built by experienced developers and for whether other indexers follow Artificial Analysis in adopting alternatives like DeepSWE. Also watch leaderboard changes as tests are swapped: the Artificial Analysis reshuffle shows how much rankings depend on the chosen benchmark.