IBM Research published VAKRA on August 12, 2026 — a benchmark that stress-tests agentic tool-use under enterprise conditions: 8,000 locally hosted, database-backed APIs across 62 domains, natural-language policy constraints, and an evaluator that replays full agent trajectories. Even the best model scores only 70.4% on single-hop tasks, 50–51% on compositional APIs, and 2.4% on policy-constrained queries it should refuse.
The benchmark has four difficulty tiers. The first two test BI-style and dashboard API interaction — the kind of tool-chaining used in business intelligence and customer support. Capability 1 spans 2,077 test instances across 54 domains, with task chains requiring 1–12 tool calls. Capability 2 surfaces a constraint: each domain exposes 6 to 328 candidate tools (average 116). The OpenAI API spec caps tool lists at 128, forcing any agent builder to implement shortlisting just to reach the upper range. That's an engineering requirement, not a quirk.
Capabilities 3 and 4 add multi-hop reasoning across structured APIs and unstructured documents. Tasks require 3–7 dependent API calls where each output must be parsed and re-parameterized into the next. The final tier adds natural-language policies — access restrictions, conditional logic — that agents must parse before invoking any tool. Performance drops over 50% as reasoning depth increases.
| Capability | Focus Area | Test Instances | Domains | Tool-Call Depth | Key Engineering Challenge |
|---|---|---|---|---|---|
| Cap 1 | BI-style API interaction | 2,077 | 54 | 1–12 calls | Multi-step task chaining across diverse domains |
| Cap 2 | Dashboard API / tool shortlisting | — | 6–328 tools per domain (avg 116) | — | OpenAI spec caps tool lists at 128 — shortlisting required |
| Cap 3 | Multi-hop structured APIs | — | — | 3–7 dependent calls | Each output must be parsed and re-parameterized into the next call |
| Cap 4 | Policy-constrained queries | — | — | 3–7 dependent calls | Natural-language access policies must be parsed before any tool invocation |
VAKRA's evaluation harness stands apart. APIs run locally via MCP servers backed by persistent databases; the evaluator replays trajectories against live tools. A three-step judge pipeline runs in order: PolicyJudge checks policy adherence, ExactMatchJudge verifies expected tool responses appear in the trajectory (order-agnostic), and a GroundednessJudge checks whether final answers root in executed outputs. The design tolerates multiple valid execution paths — crucial in enterprise workflows where no single API sequence exists.
The researchers used a fixed ReAct harness to isolate model capability from agent architecture. This makes results directly comparable across frontier and open-weight models and sidesteps prompt-engineering effects. Failure trace analysis shows breakdowns concentrate at language-mediated reasoning — entity disambiguation and cross-source grounding — not raw tool invocation. Models fail because they cannot map entities across schemas, not because they cannot call APIs.
Policy failures matter most for production readiness. A 2.4% success rate on queries that should be refused means frontier models almost always attempt to answer out-of-scope questions. In compliance contexts, that is the core failure mode, not an edge case. The public leaderboard on Hugging Face Spaces opened to external submissions in March 2026 via GitHub issue template.
Teams evaluating agentic models should run VAKRA before committing. The gap between single-hop and policy-constrained accuracy is large enough that isolated tool-calling benchmarks won't predict production behavior.