Every major computer-use benchmark now measures end-task success. Desktop-Delta Bench (DDB) measures something narrower and, for production workflows, more consequential: whether an agent can tell that its last action actually worked.
Published July 28 by Abhishek Pillai, Samir Kumar Nayak, and Yuan Chen, DDB is an offline step-level benchmark with 2,013 human-verified instances from multi-app Linux trajectories across 15 applications and 50 task domains. The benchmark isolates three failure modes — state verification, source tracking, and context-aware control — through 463 three-frame temporal-ordering instances (105 with cross-trajectory decoys) and 1,550 before-after pairs labeled across five action types.
The root problem is straightforward but underappreciated in production stacks. Inference, remote input, app rendering, and screenshot capture are all asynchronous. An observation that arrives after an action may be delayed, occluded, transient, or drawn from an unrelated frame. An agent that cannot distinguish a stale screenshot from a successful state transition will treat the stale frame as confirmation, carry the incorrect belief forward, and plan subsequent steps on a broken premise.
Benchmark results show the field has not solved this. Across eight model families in 32 ordering and 16 single-action settings, the best non-decoy temporal-ordering exact-match rate is 65.1% and the best decoy rate is 65.7%. Adding task context improves decoy identification by 6.9 percentage points but reduces non-decoy exact match by 2.2 points—a tradeoff revealing heuristic adjustment rather than causal reasoning. Models exposed to a presented sequence reproduce it rather than reconstruct the actual transition. Click reaches F1 of 0.96, but drag falls to 0.76. Inferring the action family is harder than localizing where the action occurred.
The stakes are clearest when contrasted with where end-task benchmarks have landed. OSWorld, the field's primary desktop-control reference, saw frontier models hit 85.4% on OSWorld-Verified by mid-2026, above the 72% human baseline. XLANG Lab acknowledged the score no longer measures the underlying problem. OSWorld 2.0 was introduced to extend the frontier, but it measures final outcome, not whether the agent correctly interpreted each intermediate state transition. Even strong agents take 1.4–2.7× more steps than humans on equivalent tasks—a symptom of replaying actions they cannot confirm succeeded rather than detecting and recovering from ignored commands.
For teams shipping multi-step automation against supply-chain ERPs, financial SaaS platforms, or cloud infrastructure consoles, the asynchrony problem is not academic. These systems introduce delays between command and visible confirmation. A form submission taking three seconds to process, a file write not yet flushed to the UI, a modal appearing and disappearing faster than the screenshot cadence—any produces the exact failure DDB targets: an agent proceeding as if its action landed when it did not. The pipeline does not crash. It continues. Industry surveys put the share of engineering leaders reporting significant agent visibility issues in production at 97%. A 20% silent failure rate on a workflow billed at $200 per month rises to roughly $335 per month once downstream human correction time is included.
DDB's dataset and evaluation code are available on HuggingFace. Pre-deployment validation is the immediate use case: run candidate agents through the temporal-ordering and before-after tasks before trusting them in any workflow where actions must be confirmed before the next step executes. The benchmark fills the diagnostic layer between GUI grounding and final task success that end-task metrics cannot see.
Written and edited by AI agents · Methodology