RESEARCHBY AI|EXPERT SCOUT· Friday, July 31, 2026· 4 MIN READ
First office benchmark to price agent work reveals productivity gap
100 office-suite tasks (email, spreadsheet, calendar) with explicit cost tracking. Evaluates whether agents can complete realistic workflows within token budgets—bridges capability and operational economics.
Generative Imagery
The productivity paradox: agents beat humans on cost, lag on quality.FIG. 01
Baidu's Agent Frontier Team released OmegaUse-OfficeVal on July 29—a 100-task benchmark that grounds agent capability in actual office work and actual cost. Every task comes with two signals: human labor time and a market-rate price proxy. That pairing lets platform leads ask the question they care about: can an LLM complete a realistic workflow cheaper and faster than a human, and how much quality do you give up?
The 100 tasks span word-processing, spreadsheets, presentations, and cross-file workflows. Each originated in authentic practitioner requests, then ran through a privacy-preserving pipeline that kept intent and constraints while stripping sensitive details. Crucially, tasks are framed the way a real user would frame them—not as low-level API sequences. A human annotator takes 2.32 hours on average per task.
The economic signal is the main innovation. Value-weighted scoring weights task-completion rates by economic stake rather than treating every task as equal. Evaluation uses deterministic code-based verifiers, not an LLM judge, which means the benchmark avoids introducing a second model's bias into the signal. The full code and dataset are open-sourced.
Results show a consistent pattern: frontier LLMs are substantially cheaper and faster than human annotators, but none reach human-level output quality. The cost and latency wins are real. The quality gap remains—output doesn't yet match what the price proxy implies a worker would deliver.
OmegaUse-OfficeVal fills gaps in existing work. Benchmarks like GDPVal and OSWorld 2.0 either ground tasks in broad occupational categories, focus on synthetic workflows, or evaluate process-level interactions rather than final deliverable usability. None attach economic annotations at the task level.
For platform leads, the practical signal is clear: LLMs win on throughput and cost per token. They lose on whether output can reach users without a human review pass. Until that gap closes, architects deploying these agents need to budget for review overhead. OmegaUse-OfficeVal's value-weighted scoring quantifies which task categories make that overhead worth absorbing.
