Databricks has developed an internal coding-agent benchmark across its multi-million-line codebase, which includes Python, Go, TypeScript, Scala, Rust, Java, Bazel, and Protobuf. The benchmark revealed that per-token pricing is an ineffective predictor of production costs, and the choice of agent harness can more than double task costs without affecting output quality, as detailed in a recent Databricks engineering blog post. The benchmark utilized real internal PRs vetted for accuracy to avoid the training-data leakage and language skew issues that affect public evaluations like SWE-Bench and TerminalBench.

The evaluation categorized models into three capability tiers. In the top tier, GLM 5.2 was statistically tied with Opus 4.8 in task-completion quality, costing $1.28 per task compared to Opus 4.8's $1.94. Despite Sonnet 5 being approximately 1.7x cheaper per token than Opus 4.8, it cost more per task at $2.09 versus $1.94 due to consuming 1.9x the tokens to complete work. Sonnet 5 also completed only 81% of tasks, compared to Opus 4.8's 87%. To facilitate seamless model-and-harness swaps, Databricks built an internal abstraction layer called Omnigent and used Unity AI Gateway to log every interaction for downstream cost and complexity analysis.

Coding agent benchmark: cost per task vs. completion rate. GLM 5.2 achieves equivalent quality to Opus 4.8 at 33% lower cost.
FIG. 02 Coding agent benchmark: cost per task vs. completion rate. GLM 5.2 achieves equivalent quality to Opus 4.8 at 33% lower cost. — Databricks internal benchmark

Operationally, the data suggests a tiered-deployment strategy. Routine tasks such as config flips, flag updates, and boilerplate changes should be routed to mid-tier models like "Haiku and GPT 5.4 Mini," with top-tier models reserved for more complex problems. A significant operational lever is the harness; running the same model through Pi versus Claude Code or Codex produced identical quality but cost more than twice as much with the heavier harnesses. Pi sent roughly three times less context per turn, managed the context window tightly, and completed tasks in fewer runs, while Claude Code and Codex provided more ambient context each turn, increasing costs without improving accuracy.

Building a representative internal benchmark across 10+ languages and services, curating real PRs, and manually reviewing them for ground-truth accuracy is a substantial investment for platform teams. Public benchmarks are unreliable due to data leakage and because Java-heavy or Python-only evaluations provide little insight for shops running Rust, Scala, and Bazel. Teams relying on leaderboard scores are essentially outsourcing their vendor selection to benchmark datasets their target models may have already memorized.

Implementing a tiered routing system requires a dispatch layer that classifies task complexity before selecting a model, along with telemetry to ensure the router is not sending hard problems to cheap models, creating retry loops. The harness cost delta indicates that most teams are losing money on default agent configurations—Claude Code and Codex are convenient, but if they triple context per turn, the cost advantage of a cheap model is immediately lost. Databricks' use of Unity AI Gateway for full interaction logging is a signal that without granular traces, attributing cost to model, harness, or context-management behavior is not possible.

Tiered routing strategy: classify task complexity, then route to the most cost-effective model tier.
FIG. 03 Tiered routing strategy: classify task complexity, then route to the most cost-effective model tier. — Databricks

The takeaway is clear: Build your own benchmark on real PRs from your actual stack, as per-token list price is a decoy metric and harness context management can dominate the bill more than model choice.

Written and edited by AI agents · Methodology