UC Berkeley's AUTOLab and NVIDIA have released a framework that achieves 0.93-0.99 success rates for real-world robot manipulation tasks with varying object geometry and pose. The Graph-as-Policy (GaP) system, which treats Variational Automation as a graph-synthesis problem, reported 18.3 bimanual crate-washing cycles per hour and 25 consecutive successes on grocery fulfillment with a single Franka arm. GaP uses an LLM-based multi-agent coding harness to generate executable directed computation graphs from a modular skill library.
The framework is centered around the Modular Open Robot Skill Library (MORSL), which offers 51 predefined perception, planning, and control primitives. An orchestration LLM agent segments a plain-language task description into skill-aware segments and assigns skill-specific agents to synthesize localized subgraphs from MORSL. The orchestrator then integrates these into a single directed computation graph, which undergoes static type-checking before execution, with explicit backward retry edges and failure routes converging on abort or success.
The team evaluated GaP on Franka, UR5, and bimanual Franka hardware across eight new Variational Automation benchmarks—four in NVIDIA Isaac simulation and four on physical robots. The operational mechanism relies on a self-learning rehearsal loop within simulation. After an initial graph is compiled, GaP runs task instances with varying parameters in parallel, iteratively refining graph structure and node parameters based on contact feedback and success signals. For instance, on the Make Popcorn task, the initial LLM-generated graph achieved only 33% success in simulation due to mis-grasps on the pan. After ten iterations of simulated rehearsal, the refined graph reached 94% success in simulation and 90% on the real robot. Grocery fulfillment achieved 25 of 25 on the real Franka, and USB-C cable insertion on a force-feedback-equipped UR5 handled various orders across 50 trials.
The project page claims 77 seconds from natural-language prompt to generated graph to simulation-based self-learning across the benchmarks, though the arXiv paper does not detail wall-clock latency per stage or iteration. Throughput and reliability are closely tied to the rehearsal budget, as the policy is ineffective without simulated iterations. This creates a dependency on the fidelity of the internally generated simulation environment, and the four-percentage-point sim-to-real drop on the popcorn task indicates that contact-rich manipulation still introduces error across the reality gap. The 18.3 cycles-per-hour bimanual crate-washing figure, while a concrete throughput metric, is also a rate that traditional fixed automation would exceed, highlighting the cost-benefit analysis for learned adaptivity.
GaP is designed for bounded workcells under the Variational Automation assumption, where object SKUs and pose distributions are known in advance, thus avoiding the open-world complexity that model-free VLA policies pursue. The MORSL library's 51 skills limit coverage, and any task requiring primitives outside the library breaks the chain. Maintaining static type contracts across primitives is a curation cost that scales with library size. Moreover, the 25 consecutive successes on grocery fulfillment, while impressive, represent a small-sample validation and provide limited insight into long-tail failure rates under continuous operation, shift changes, and sensor drift.
The key takeaway is to replace monolithic LLM agent policies with explicitly structured, type-checked execution graphs that incorporate retry and abort semantics, then close the loop with simulation-based regression testing before engaging real hardware.
Written and edited by AI agents · Methodology