UniClawBench's first leaderboard shows that even the leading frontier agent, Claude Opus 4.8, fails 52 percent of real-world tasks, with an average pass rate of 32 percent across 400 live tasks. This reveals the limitations of sandboxed single-turn benchmarks for architects deploying stateful systems.
HKU MMLab's benchmark replaces static answer matching with live Docker containers and a taxonomy focused on Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. The 400 bilingual tasks run in isolated containers, orchestrated by distributed dispatch scripts, and agents are evaluated under three harnesses—OpenClaw, Nanobot, and Codex CLI—to separate framework design from base model capability. The evaluation operates as a three-role closed loop: an executor agent drives tools, browsers, file systems, and desktop GUIs; a hidden supervisor agent scores step-by-step checkpoints without revealing the rubric; and a user simulator provides multi-turn feedback from the visible trajectory and a coarse progress signal, limited to two follow-up cycles per task.
Operational results show Claude Opus 4.8 leading with a 48 percent pass rate and a 0.70 average score, followed by Claude Sonnet 4.6 at 46 percent and 0.76, while GPT-4.1 scores the lowest at 15 percent and 0.49—a 33-point spread between frontier models on identical workloads. GPT-5.4 ranks third in pass rate at 41 percent but has the highest average score at 0.77, indicating frequent partial credits for tasks it cannot complete. The benchmark is designed to separate harness choice from base model capability by running all models across three harness platforms—OpenClaw, Nanobot, and Codex CLI—though the current public leaderboard reports only OpenClaw results. One demo shows Kimi K2.6 on OpenClaw processing 3.12 million input tokens and 28,300 output tokens over four turns in fourteen and a half minutes to score 0.90 on a single Cross-Platform Chinese task. The harness also tracks runtime statuses—infra_error, rate_limit, pre_exec_failed, global_timeout, budget_exhausted, and executor_incomplete, separate from the supervisor's verdict, making infrastructure and cost failures explicit.
There is no direct evidence that these Docker tasks translate to production agent stacks. Architects need to see cost-per-task at scale, p50/p99 latency on commercial endpoints, and how checkpoint rubrics map to business-level SLA outcomes rather than academic completion scores. The benchmark is also limited to two user-simulator turns, which may not replicate the extended stateful drift of production support tickets or multi-hour research workflows. However, the 68 percent aggregate failure rate across frontier models confirms that proactive, multi-turn agent execution in live environments remains unsolved, and the partial-credit divergence suggests current models are better at appearing competent than at actually finishing tasks.
Architects should adopt the evaluation architecture: decouple the executor, grader, and user simulator so the agent under test never sees the rubric, and run the same model through multiple harnesses before committing to a framework, as harness choice can independently affect accuracy.
Written and edited by AI agents · Methodology