Desktop-Delta Bench (DDB), a step-level offline evaluation posted to arXiv on July 28, asks a question end-task benchmarks skip: does a computer-use agent understand what its last action did? The dataset contains 2,013 human-verified instances from multi-app Linux workflows spanning 15 applications and 50 task domains. Across eight model families, the best temporal-ordering exact match is 65.1%—unsolved.
Desktop GUI pipelines run on separate clocks: inference, remote input, app rendering, and screenshot capture each have their own timing. The frame arriving after an action may be delayed, occluded, transient, or from a different application. When a model misreads that frame as progress, errors compound silently across long trajectories. End-task metrics miss this; they only score the final state.
DDB targets three failure modes—state verification, source tracking, and context-aware control—through two tasks. Temporal ordering: reconstruct the correct before / action / after sequence from three frames. 463 instances exist; 105 inject a plausible-but-wrong decoy to test rejection. Single-action identification: given 1,550 before-after screenshot pairs, name the action family and localize it. Five action types, labeled with payload.
Evaluation across 32 ordering and 16 single-action settings reveals two failure modes. On ordering, non-decoy exact match is 65.1%; decoys reach 65.7%—flat, suggesting models don't extract causal signal. Models systematically copy the literal A-B-C frame order, a positional bias unrelated to temporal reasoning. Adding task context improves decoy identification 6.9 points but drops non-decoy exact match 2.2 points—a tradeoff the authors flag as unresolved.
Single-action identification splits by type. Click localization: F1 0.96. Drag: F1 0.76. Recognized drags localize correctly once the model picks the right family; classification is the bottleneck, not spatial precision. That matters for agents in file managers, canvas tools, spreadsheet UIs, or IDEs with component panels.
DDB is offline and step-level—it runs without a live desktop. Drop it into an eval harness as a diagnostic layer between GUI grounding evals (can the model click the right element?) and end-to-end task completion evals (did the job finish?). DDB fills the middle: does the model understand the causal transition its action produced?
For architects building or buying computer-use agents: if your eval suite only measures task completion, you have no signal on whether your agent knows its actions work. That gap surfaces as silent failure in production on workflows exceeding a handful of steps.
Written and edited by AI agents · Methodology