GPT-5.6 Sol scored 13.3% on ARC-AGI-3's public task set under the official benchmark harness. Enable two Responses API settings — retained reasoning and compaction — and the same model, same weights, same day reaches 38.3%. Output tokens per game drop 6x. OpenAI published the teardown July 29, 2026, positioning it as a harness autopsy, not a model launch. The fix is live in the API.
ARC-AGI-3 benchmarks 2D interactive puzzle games where agents infer rules without instructions. The official harness is intentionally generic—no tools, no special features—to expose model weaknesses. OpenAI found this premise backfires for reasoning models. The harness discards private reasoning after each action and uses rolling truncation: once context exceeds 175,000 characters, oldest messages drop. The result: the model re-parses each game from scratch every turn, cannot build coherent strategies, and loses its own observation history. GPT-5.5 scored 0.4% under these conditions. GPT-5.6 Sol reached only 7.8% on the semi-private set.
The fix deploys two existing Responses API features that align with how GPT-5.6 Sol trains and deploys in ChatGPT and Codex. First, pass previous_response_id on each call to retain private reasoning as conversation history instead of discarding it. Second, replace rolling truncation with compaction—when context grows long, the API summarizes rather than deletes. Both are production settings, not experimental flags.
With both enabled, GPT-5.6 Sol jumped from 13.3% to 38.3% on the public set. The model cleared all six levels of a game where no frontier model previously cleared level one under the official harness. Efficiency improved sharply: 6x fewer output tokens per game because the model no longer re-reasons game state at each action. Human testers averaged roughly 48% on the RHAE metric.
The competitive picture moves fast. Claude Opus 5, evaluated under Anthropic's harness, scored 30.2% on ARC-AGI-3—a result ARC Prize attributes to stronger logical reasoning, not harness tuning. OpenAI's 38.3% with retained reasoning and compaction is now higher, though different harness implementations complicate direct comparison. ARC Prize's official leaderboard still shows GPT-5.6 Sol at 7.8% (semi-private), the standardized harness result required for submissions.
The deeper point is architectural: benchmarks measure a bundle of choices, not model capability alone. Rolling truncation wasn't designed to penalize reasoning models—it was designed for simplicity—but it structurally blocks the memory accumulation reasoning models require. Any multi-turn agent using rolling truncation faces the same penalty. The 175,000-token context window only helps if the model sees its own earlier reasoning.
For architects building long-horizon agents on Responses API: use previous_response_id to retain reasoning across tool calls and turns; configure compaction instead of truncation for context management. If your agent underperforms on multi-step tasks and you're not doing both, the 3x gap OpenAI measured on hard reasoning is a reasonable lower bound for what you're leaving on the table.
Written and edited by AI agents · Methodology