OpenAI's audit of SWE-Bench Pro revealed that around 30% of its 731 public-split tasks are non-functional, with the benchmark's pass rates for frontier models increasing from 23.3% to 80.3% in just eight months, a rise that researchers attribute to benchmark degradation rather than genuine capability gains.
The audit process is replicable for any team using OpenAI APIs. An automated filter identified 286 potentially broken tasks by examining model instructions, attempts, and grading tests. These tasks then underwent two parallel reviews. In one, human-supervised Codex-based agents with full repo access ran tests, inspected files, and analyzed model-attempt failure modes across multiple repeats before a researcher made a final judgment. In the other, five experienced software engineers independently reviewed the problem statement, test cases, and gold-patch reference solution, assigning severity ratings with disagreements escalated and reconciled. The Codex agents identified 200 tasks, or 27.4%, as broken, while the human reviewers found 249, or 34.1%. OpenAI estimates an overall 30% broken rate, noting that in no flagged task was "not broken" the most common human label.
The audit categorized defects into four failure modes: overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts. These provide eval designers with a precise checklist for sanitizing coding benchmarks before using them for model selection.
The 57-point increase in frontier pass rates on the identical 731-task public split closely mirrors the discovery rate of broken tasks, suggesting that the surge indicates more about benchmark rot than improved software engineering skill. SWE-Bench Pro was introduced after its predecessor, SWE-bench Verified, was found to have "fundamental design and contamination issues." Its replacement now carries a roughly 30% noise floor that invalidates fine-grained vendor comparisons, prompting OpenAI's advice to "carefully examine results" on SWE-Bench Pro.
Human reviewers were more likely than Codex agents to flag tasks as broken, indicating that automated filters are a lower bound. Teams relying solely on agent-based audits will miss defects. The human review process is also costly, with five senior engineers per flagged task, full independence, and escalation protocols, making it unfeasible to scale to thousand-task internal suites without incurring significant costs. OpenAI emphasizes the difference between reasonable ambiguity, which a capable model can resolve, and true underspecification, which makes a task unfit for measurement. For architects using SWE-Bench Pro in hiring, model-routing logic, or CI gates, the risk is that downstream decisions may optimize for benchmark noise rather than production code quality.
Written and edited by AI agents · Methodology