QuoteBench Exposes 55-Point Gap Hidden in Coding-Agent Benchmarks

A new benchmark published yesterday isolates a failure mode standard coding-agent evals have been hiding in plain sight: the gap between what an LLM generates and what the shell actually runs. QuoteBench, from researchers at LMU Munich and Zhejiang University, tests 56 one-shot Bash tasks across 14 incident-derived families in eight harness configurations. A deliberately unescaped parser between generation and execution drops task success by 55.4 to 73.2 percentage points depending on configuration.

The paper distinguishes three components: the generation contract (what the model emits), the execution transport (how the harness serializes, wraps, and reparses that output), and the final-state validator (whether the eval checks actual filesystem state or just matched strings). Current benchmarks collapse all three. A "matched score" tells you whether the command looked right; it does not tell you whether the same command survives a real execution path. QuoteBench introduces an unescaped parser representing the serialization layers common in agent frameworks — JSON wrapping, subprocess string handling, MCP transport — and measures what happens to each task when the same model reply passes through it.

The results expose two separate phenomena. First, damage: the unescaped parser alone accounts for 55.4 to 73.2 percentage points of task-success loss without changing a single character of the model's generation. Second, compensation: when models are told about the execution boundary, they adapt their generation style and recover 30.4 to 60.7 percentage points on six of the eight configurations — but zero or slightly negative on the other two. Some harness-model combinations cannot adapt at all, even with explicit prompt disclosure.

GPT-5.6-sol exposes this hidden structure. Its matched gap — the headline number a conventional benchmark would report — is −3.6 percentage points. The QuoteBench accounting underneath: −64.3 points of damage from the parser and +60.7 points of compensation from the model's generation change. The matched score is arithmetically correct and operationally meaningless. A team that picked GPT-5.6-sol because it looked nearly equal to the field on a matched-score leaderboard is running a model that depends on 60.7 percentage points of self-correction just to stay competitive on that configuration.

Ranking instability matters for model selection. Across 26 comparable model pairs, the deployment configuration produces one unambiguous model reversal and four more that sit on single-task margins. If you chose a model from a matched-score leaderboard and then deployed it through a harness with different serialization semantics, you may be running a suboptimal model for your stack — and you would not know it from the benchmark number.

The production CI/CD implication is direct. Teams running coding agents in unattended pipelines — the --auto-exec mode that security researchers have documented as the dominant real-world deployment pattern — have no disclosure mechanism. The model receives no information about the execution transport, sits in the worst-case compensation zone, and the final-state validator on most evals never checks whether the filesystem ended in the right state.

The QuoteBench paper recommends a disclosure requirement, not a model ranking: any coding-agent evaluation should report model configuration, generation contract, execution path, operating point, and a final-state validator. A single matched score is not an intrinsic model property. It is a property of a specific model-harness pair, and the harness is half the system shipping to your CI runners.