GPT-5.6 Sol tripled ARC-AGI-3 score to 38.3% with retained reasoning + compaction

OpenAI discovered that GPT-5.6 Sol's initial 7.8% performance on the ARC-AGI-3 benchmark (a test of 2D puzzle reasoning) was not a model limitation, but a harness design issue. By enabling two API settings—retained reasoning and compaction—the model's score jumped to 38.3% on the public task set, and importantly, GPT-5.6 Sol now solves all six benchmark levels. The same settings are already in production in ChatGPT and Codex but were missing from ARC's generic evaluation harness.

The core insight: models trained with internal reasoning chains perform poorly when each new action discards that reasoning and rolling truncation loses earlier observations. By retaining reasoning across turns and replacing rolling truncation with compaction (which summarizes instead of deletes old context), GPT-5.6 Sol spent less time thinking per action and learned coherent strategies over time. OpenAI's Responses API made the fix simple: passing a previous response ID automatically carries forward reasoning. Output tokens were cut 6x while maintaining better inference trajectories.

For practitioners evaluating frontier models on agentic benchmarks, this illustrates a critical confound: benchmark harness design can mask or reveal model capability. Models that appear weak in minimal, generic harnesses may excel under production-like setups with stateful reasoning and context management. The ARC-AGI-3 result suggests that long-horizon reasoning tasks—multi-turn code generation, planning in simulation, game-playing—may benefit more from retained-reasoning patterns than raw scaling.