Databricks has open-sourced OfficeQA Pro V2, a grounded-reasoning benchmark built from 120,000 pages of U.S. Treasury financial records spanning 233 years. The benchmark contains 90 questions drawn from a corpus previously unseen by any system under test—forcing generalization rather than rewarding memorization.
The motivation is direct: frontier models improved substantially on the original OfficeQA Pro over seven months. That raised a hard question—were gains evidence of better reasoning, or overfitting to a single corpus? Databricks tested it via the Grounded Reasoning Cup, a live event where 11 academic teams backed by OpenAI, Anthropic, and Google DeepMind built specialized agents against an undisclosed corpus. The benchmark is now public.
Default provider harnesses—Claude Code for Anthropic, Codex for OpenAI—averaged 26.0% accuracy across five models. Sonnet 5 scored 15.6% at $5.01 per rollout; GPT-5.6 Sol hit 33.3% at $4.70. Higher spend did not ensure higher accuracy. Purpose-built competition agents averaged 41.1%, with the winning team reaching 63.3%.
| Configuration | Harness / Type | Accuracy | Cost per Rollout |
|---|---|---|---|
| Default harnesses (5-model avg) | Provider default | 26.0% | — |
| Sonnet 5 | Claude Code (Anthropic) | 15.6% | $5.01 |
| GPT-5.6 Sol | Codex (OpenAI) | 33.3% | $4.70 |
| Competition agents (11-team avg) | Purpose-built | 41.1% | — |
| Winning team | Purpose-built | 63.3% | — |
The larger finding: agent harness matters more than the underlying model. Running frontier models through Databricks Genie—which pre-parses the corpus with ai_parse before any agent interaction—improved accuracy by 24.0 points on average, a 92% relative gain. Across four direct model comparisons, mean accuracy shifted from 37.5% to 52.8%; the strongest Genie configuration reached 60%. Claude Fable 5 is illustrative: default Claude Code averaged $37.36 per rollout and looped through repeated parse attempts; switching to Genie cut cost by 9x while adding 14.4 points of accuracy. On the cost-quality frontier, Genie configurations with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Fable 5 dominated—better accuracy and lower cost than provider defaults.
| Configuration | Harness | Accuracy | Cost per Rollout |
|---|---|---|---|
| 4-model mean | Default (Claude Code / Codex) | 37.5% | — |
| 4-model mean | Databricks Genie | 52.8% (+24.0 pp) | — |
| Best Genie configuration | Databricks Genie | 60.0% | — |
| Claude Fable 5 | Claude Code (default) | baseline | $37.36 |
| Claude Fable 5 | Databricks Genie | +14.4 pp vs. default | ~$4.15 (9× cheaper) |
Persistent failure modes mirror the original benchmark: parsing fidelity errors, incorrect temporal reconciliation across fiscal years, and multi-hop reasoning chains breaking when evidence spans non-contiguous sections. These are not edge cases—they match the failure patterns in mixed-vintage, inconsistently formatted document collections common in enterprise environments.
| Failure Mode | Description | Enterprise Relevance |
|---|---|---|
| Parsing fidelity errors | Incorrect extraction of text or figures from source PDFs | Common in mixed-format, multi-vintage document collections |
| Temporal reconciliation errors | Incorrect alignment of values across fiscal years | Financial and regulatory records spanning multiple reporting periods |
| Multi-hop chain failures | Reasoning breaks when evidence spans non-contiguous document sections | Complex queries requiring synthesis across dispersed sources |
The benchmark uses deterministic exact-match scoring with 0.0% tolerance, the same approach as OfficeQA Pro. That strictness avoids inflation from partial credit and keeps results comparable across configurations. The corpus—U.S. Treasury Accounts of Receipts and Expenditures, released for the 250th anniversary—is publicly available for reproduction without a Databricks account.
For architects evaluating RAG or document-agent stacks, the benchmark is available for self-evaluation. The Genie results demonstrate that pre-parsing infrastructure is first-order, not an optimization detail. A 60% ceiling on 90 questions built from government PDFs should temper confidence in any system promoted for production document QA.