Apple Research has introduced MM-ToolSandBox, a benchmark and execution environment designed for visually grounded tool-calling agents. The initial results are striking: across 500+ tools and 258 human-verified scenarios, even the best frontier model failed more than half of its tasks. The framework assesses multi-turn, multi-image agent workflows, where visual inputs are received progressively and the agent must ground them into executable tool calls while managing state mutations, goal revisions, and error corrections in a stateful runtime.
The stack extends ToolSandbox and AppWorld with visual grounding across 16 application domains, supporting both function-calling and code-execution interfaces. The evaluation harness uses a dual-mode judge: rubric-based LLM grading alongside static entity-diff frameworks. The repository includes parallel execution support up to 16 workers, a local visualization server on localhost:8000, and full trajectory logging for post-hoc analysis. Setup requires Python 3.11+, approximately 7 GB of disk space for scenario images, and API keys for OpenAI, Anthropic, and Serper, indicating an external-API-dependent pipeline rather than a self-contained offline harness.
The scenario corpus is divided into two sets: 258 nominal multi-app, multi-turn tasks for the full benchmark, and 50 UI-rendering scenarios for an A2UI interactive surface. An automated generation pipeline produces candidates through information-flow-guided planning and multi-stage filtering, with the nominal set being human-verified. The GitHub README provides example runs using GPT-5 as the agent and Claude 4.5 Sonnet as the judge, reflecting an evaluation pattern that pairs frontier models in adversarial or referee roles and ties benchmarking costs to the price of multiple frontier APIs.
The results underscore the immaturity of visual tool-calling. The arXiv paper evaluated twelve models ranging from 4B open-weight checkpoints to top-tier proprietary systems, with none achieving a 50% success rate. Failure analysis reveals that 53% of failures resulted from incorrect information extraction from images, even when the agent had otherwise constructed a correct task workflow. The error mode shifts with model scale, complicating capacity planning—smaller models fail at planning, while larger models fail at visual precision. This suggests that scaling alone will not close the perception gap, and teams may need to combine small planners with large vision backbones rather than relying on a single monolithic model.
This release is a research benchmark with no evidence of production agent deployment. For architects considering adoption, the practical friction points are clear. The LLM-as-judge setup introduces evaluator bias, variance, and cost—each benchmark run consumes tokens across multiple provider APIs with no guarantee the judge is more visually reliable than the agent under test. The 7 GB image payload and multi-turn stateful execution make this a heavy integration suite that will consume significant GPU-hours and API budget. The framework also does not address how these agents behave under real rate limits, concurrency caps, or user-facing latency budgets, all of which are critical for a visual tool-caller to be deployable.
For internal replication, architects should consider adopting the stateful sandbox with full trajectory replay, parallel worker support, and static entity-diff scoring, replacing the LLM judge with deterministic or domain-specific scorers wherever task permits.
Written and edited by AI agents · Methodology