A study from UC Berkeley and Arena Intelligence quantifies how much the evaluation harness itself influences coding agent performance, isolating harness effects from model capability. The researchers evaluated 21 model–harness pairs spanning seven models (Claude Fable 5, Claude Opus 4.8, Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.6 Sol, GPT-5.6 Luna, and Kimi K3) and three harnesses—Claude Code, Codex CLI, and Pi—on SWE-bench Lite and Terminal-Bench 2.0. The finding that matters most to architects: harness choice can drive a 5x cost difference while success rates stay nearly flat.
The study reveals that the same model often achieves similar success rates at substantially different costs. Claude Fable 5, for instance, solves 97.8% of attempts in Claude Code, 96.7% in Codex, and 96.7% in Pi, yet Claude Code costs about twice as much as Pi ($1.33 versus $0.67 per rollout). Across shared models on SWE-bench Lite, Claude Code costs about 2.0x as much as Pi and 1.6x as much as Codex, using geometric means of cost ratios. Meanwhile, the average harness effect on success rate stays within ±2% on SWE-bench Lite and within about ±5% on Terminal-Bench 2.0. The researchers call this hidden cost difference the "harness tax"—paying extra for essentially the same quality because of harness choice alone.
The cost gap originates in harness design itself. Claude Code's mean initial context is over 10x Pi's, with longer instructions and larger tool schemas. For Claude Fable 5 on SWE-bench Lite, Pi and Claude Code average 15.4 and 15.3 turns per attempt respectively, yet Claude Code spends about twice as much per turn. Pi reaches the Pareto frontier on both benchmarks by providing just four tools: read, write, edit, and bash. This demonstrates that a minimal, open-source harness can be competitive on both cost and task success rate, opening opportunities for harness research without access to proprietary systems or co-training with models.
A second finding challenges the assumption that models perform best with their own harness. Across six Anthropic and OpenAI models and both benchmarks, an alternative harness achieves the highest observed success rate in nine of twelve comparisons. Claude Sonnet 4.6 solves 68.9% of attempts in Codex versus 66.7% in Claude Code on SWE-bench Lite at similar cost. GPT-5.6 Sol, which OpenAI describes as optimized for agentic software engineering in Codex, achieves an 83.3% success rate in Pi versus 78.9% in Codex on Terminal-Bench 2.0, at about half the cost ($0.42 versus $0.76). These results show that a model's capabilities are compatible and generalizable across harnesses, and that provider optimization does not guarantee the best pairing.
The study is limited to two open-source benchmarks that models may have encountered during training, and results may differ on other benchmarks and workloads. The researchers note that harness selection becomes more pressing for coding agents given the volume and spread of their usage, and that the natural next step is to evaluate harnesses and automate their selection in real development workflows where requirements evolve and tasks extend across sessions.
For architects building or comparing agent eval frameworks, the takeaway is direct: measure cost and success rate across harnesses for the same model, because harness design can dominate the signal you're measuring and hide substantial cost differences behind similar task success.