Databricks Cuts Agentic AI Code Costs 50 Percent with Caching and Routing

Databricks published a cross-company cost-management guide for org-wide AI coding deployments, drawing on data and input from Stripe, Coinbase, Uber, and Ramp. The core problem is structural: consumption-based pricing scales with output, not headcount, and agentic coding multiplies token volume per engineer faster than traditional seat-cost models can track. At Databricks, agentic coding improved every velocity metric and produced order-of-magnitude output gains on some teams. Spend scaled alongside.

The guide organizes cost control around the "efficiency frontier" — the set of models offering the best price per unit of intelligence, distinct from the raw intelligence frontier labs chase. This frontier moves fast: new models releasing weekly offer better price/performance than incumbents. Public benchmarks are poor proxies for real-world coding tasks. Databricks built internal evals, found GLM models competitive, and rolled them out. Stripe tested Opus 4.7, found it did not meaningfully improve on Opus 4.6 at higher cost, and blocked the rollout. Databricks saw cost regressions comparing Opus 5.0 to 4.8.

Model flexibility faces harness lock-in. Proprietary frontier models are co-designed to work with specific harnesses, so switching models often means switching the developer's entire toolchain. The guide recommends meta-harnesses: a layer surfacing consistent developer UX while dispatching requests to underlying harnesses — proprietary or open source — based on task type and cost. Databricks open-sourced its meta-harness, Omnigent, and is shipping a managed version on Databricks under Unity AI Gateway governance.

Routing adds a second lever. At the request level, a stateful proxy directs each inference to the lowest-cost capable model, accounting for server-side cache state. At the task level, the meta-harness classifies complexity — a rename operation versus an open-ended architecture question — and dispatches the task to the appropriate tier before tokens are exchanged. Unity AI Gateway's Smart Router operates at request level and cuts average task cost by over 30%.

Prompt caching delivers highest-return low-infrastructure gains. Cache writes cost money, but cached reads reduce per-inference spend substantially on repeated-context workloads. Tuning harness and caching settings internally gave Databricks a 50% reduction in generated tokens with no observed quality loss. The main operational knob is cache TTL and hit-rate tuning by workload type.

Budget architecture is where most organizations fail. Hard per-user spending caps block the wrong engineers — productive ones running long agentic loops — while ignoring actual waste: accidental multi-agent spawns on Friday nights or retry loops that multiply cost 10× overnight. At Databricks' scale, 500 to 1,000 engineers hit hard limits monthly, flooding the internal #ai-devtools Slack channel with unblock requests. The replacement design pairs daily and monthly budgets at a fixed ratio, so steady spenders never trip the daily limit, and uses self-serve acknowledgements for daily overruns rather than approval queues. Unity AI Gateway's spend controls layer this across user, use case, workspace, and account. Databricks' public guidance offers examples: $2,000/user/month per-engineer budgets, $1,000/user/month alerts for coding-agent workloads, $50,000/month for production workspaces, and $200,000/month org-level ceilings — though the company notes these are not its actual internal numbers.

The practical sequence: lock model flexibility first via meta-harness or gateway routing, tune caches, then add routing logic, then replace hard caps with layered progressive budgets. Organizations without centralized observability across coding tools have no data to optimize against. Unity AI Gateway passed a quadrillion tokens in its first year, giving Databricks' platform team the signal to iterate.