Stanford's agent won Databricks' inaugural Grounded Reasoning Cup with 63.3% accuracy on OfficeQA Pro V2, a benchmark built from 120,000 pages of U.S. Treasury financial documents. Out-of-the-box frontier agents averaged below 30% on the same benchmark. System design beat model choice.
The competition pitted 11 academic teams from the U.S. and Canada against each other under deliberately adversarial conditions. Each team partnered with OpenAI, Anthropic, or Google DeepMind and was locked to that provider's model family. Teams spent two months optimizing against OfficeQA, Databricks' grounded-reasoning benchmark. On competition day they received a new corpus: the U.S. Treasury's Accounts of Receipts and Expenditures, released 36 hours prior. The short window forced teams to ingest and index the data without overfitting.
Six 15-minute rounds, 15 questions each. Difficulty escalated; final-round questions doubled in weight. The first correct answer per question earned a 0.25-point speed bonus, measuring both accuracy and latency.
| Dimension | Detail |
|---|---|
| Rounds | 6 |
| Questions per round | 15 |
| Total questions | 90 |
| Final-round weight | 2× (double points) |
| Speed bonus | +0.25 pts — first correct answer per question |
| Difficulty | Escalated across rounds |
Stanford won. UMass Amherst and Yale took second and third. All three finished 20+ points clear of the field. Teams using the same underlying model differed by an average of 30.4 points, isolating system design as the primary lever. Top teams shared four tactics: document preprocessing before retrieval, parallel agent execution, structured tool use with explicit verification, and targeted fallbacks when primary document-representation methods failed. Stanford packaged these into a skill library with adaptive answer verification.
| Tactic | What it does | Stanford addition |
|---|---|---|
| Document preprocessing before retrieval | Cleans and structures pages before chunking/indexing | — |
| Parallel agent execution | Multiple agent lanes run concurrently to improve throughput | — |
| Structured tool use + explicit verification | Each tool call is followed by a formal check step | Adaptive answer verification via skill library |
| Targeted fallbacks | Alternative document-representation method fires when primary fails | — |
Across all 90 competition questions, 18.8% went unsolved by every team. These represent systematic retrieval and parsing gaps no architecture closed. Stanford's winning system, which outperformed baseline frontier agents by 35 points, still answered only 63.3% correctly. Enterprise grounded reasoning over complex financial corpora remains unsolved.
Benchmark-driven agent development surfaces an old problem: techniques tuned to a known corpus don't generalize reliably. Approaches strong on OfficeQA degraded on OfficeQA Pro V2's different document structure and vocabulary. The held-out, same-day corpus made this failure visible. Static benchmarks don't force this test.
For production RAG pipelines and document-QA systems, Grounded Reasoning Cup results suggest a concrete checklist. Retrieval fallbacks — not single strategies — are structural. Parallel agent lanes with separate verification improve throughput and confidence. Latency is observable and carries cost: the speed bonus models production queuing. Model swaps within the same provider shift scores by points; system-level changes move them by tens.
The OfficeQA benchmark suite is public. Practitioners building enterprise document agents now have a reproducible test for the failure modes the Grounded Reasoning Cup exposed.