Databricks held the inaugural Grounded Reasoning Cup, a live agent competition testing 11 academic teams on their ability to reason over enterprise document collections. Stanford's team won with 63.3% accuracy on a previously unseen benchmark (120,000 pages of U.S. Treasury documents), outperforming the average frontier agent baseline (under 30%) by 35 points and other teams by 22 points. The competition ran six 15-minute rounds with progressively harder questions, with teams having only 36 hours to index and prepare on the new corpus.
The key finding: benchmark generalization is fragile. Teams developed and optimized on OfficeQA but found improvements did not reliably transfer to the new task. The average gap between the best and worst team using the same frontier model was 30.4 points—underscoring that system architecture (parsing, retrieval, verification, tool composition, parallelism) matters as much as the underlying LLM. Eighteen percent of all questions went unsolved by every team, indicating substantial room for continued improvement in enterprise document reasoning.
Stanford's winning approach combined a library of reusable skills, targeted fallback strategies for different document representations, parallel agent decomposition, structured tool use, and end-to-end verification loops. The results stress that enterprise grounded reasoning is not a pure model-capability problem; it requires careful orchestration of retrieval, parsing, tool-use patterns, and validation.
This pattern aligns with broader signals: frontier agents exhibit strong zero-shot capability but require substantial tuning for domain generalization. For architects building production document-reasoning systems or enterprise search applications, the 30-point skill gap and 18% unsolved rate suggest evaluation on held-out domains and multi-agent orchestration are essential before deployment.