4 cost levers Databricks uses to control AI coding spend
AI coding tools are genuinely productive. At Databricks, agentic coding has improved every velocity metric they track, with some teams seeing order-of-magnitude gains in output. But there is a catch that every company deploying these tools at scale eventually hits: costs grow exponentially. Left unchecked, that curve will eventually overtake the efficiency gains AI provides.
Databricks, along with infrastructure teams at Stripe, Coinbase, Uber, and Ramp, has converged on a shared playbook. The goal is a dual mandate: broad, low-friction access to AI tooling for every developer, kept inside a roughly fixed cost envelope per user. This post breaks down the four cost levers they have identified, based on Databricks’ internal experience and informal conversations across those companies.
Why the efficiency frontier matters more than the intelligence frontier
The term frontier model typically means the highest-intelligence model available. But when AI runs at scale across an engineering org, a different concept matters more: the efficiency frontier. The efficiency frontier is defined by the set of models that offer the best price point for a given level of intelligence.
Most day-to-day coding does not require solving novel math proofs or discovering zero-day exploits. What matters in aggregate is the cost of models that clear the quality bar for typical software engineering work. That efficiency frontier is advancing far faster than the intelligence frontier, with new models releasing almost weekly that offer better intelligence per dollar than prior options.
️ Cost lever 1: move to open source and lower-cost models
Switching to newer, more efficient models as they release is the single largest cost lever available. But capturing those gains requires knowing which models actually beat your incumbent on real work, not public benchmarks.
Several companies have built internal automated evaluations they consider more representative of their actual development mix. Databricks published an example benchmark against their multi-million-line codebase and observed highly competitive price-to-performance for GLM models, which they subsequently rolled out internally.
Evaluations frequently produce negative results too. Stripe found that Opus 4.7 did not meaningfully improve quality over Opus 4.6 while increasing cost, so they declined to make Opus 4.7 available internally. Databricks saw similar cost regressions when comparing Opus 5.0 to 4.8. The point: chase the efficiency frontier with your own data, not vendor benchmarks.
Cost lever 2: dynamic request and task routing
Rather than asking developers to manually pick the right model for each task, automatic routing can squeeze further efficiency out of agentic workflows. Three routing patterns have emerged:
- Request-level routing: A stateful proxy sits between the coding harness and the underlying models, routing each inference request to the lowest-cost model capable of answering it. Routing for agentic use cases also needs to account for server-side caching, since cold cache hits are expensive for large context workloads. Products like Cursor Router, OpenRouter’s AutoRouter, Ramp’s Router feature, and Databricks’ Smart Routing in Unity AI Gateway are examples of this approach.
- Task-level routing via meta-harness: A client-side process dispatches entire user tasks to different harnesses based on task complexity. A simple task like renaming a component goes to a cheap model. An open-ended architectural question goes to a more capable one. Omnigent is an example meta-harness that supports this pattern.
- Escalation and delegation patterns: A single harness pairs two models. In Claude’s Advisor Tool, the cheaper model runs the main loop and escalates when it needs more capability. Cognition’s Devin Fusion inverts this: the higher-cost model runs the main loop and delegates work to a cheaper model selectively.
Databricks reports that their Unity AI Gateway Smart Router consistently reduces average task cost by more than 30% while roughly matching the quality of the most expensive model in the working set. Other companies they spoke with reported similar results.
️ Cost lever 3: visibility, spend gates, and progressive friction
Hard budgets, where access is cut off at a token spend threshold, are used only as a last resort at every company in this group. Two reasons: first, cutting off a developer mid-work is debilitating to productivity. Second, some of the highest spenders are also the most productive developers, generating the most output. Penalizing them is self-defeating.
Instead, the pattern that has emerged is progressive friction:
- Visibility: Every company provides near-real-time spend feedback to developers across all tools, along with tips for reducing spend by switching to lower-cost models.
- Spend gates: Developers are prompted to take action or seek approval as spend increases. The simplest version is a self-clearing warning that spend is trending above a threshold. Databricks found self-clearing gates effective at catching accidental spend. Further gates can require explicit budget approval through a management chain.
- Downshifting: When a developer hits a spend gate, they get routed to a lower-cost model rather than suspended entirely. Since the lowest-cost models are drastically cheaper than frontier-intelligence models, developers keep working without accruing massive ongoing spend.
- Suspension: Full token access suspension exists as a limit case but is treated as a temporary measure and a starting point for a conversation about efficient AI use.
✂️ Cost lever 4: reduce token overhead
When a developer types a simple request like “investigate and fix this bug,” the AI agent then gathers context, invokes tools, searches the codebase, and loads system information. By the time LLM inference runs, the developer’s original message is a negligible fraction of the total token input. Context overhead dominates cost.
Several approaches are being explored to reduce that overhead:
- Forcing more frequent compaction (compression) of active context.
- Using harnesses that generate less token overhead, or tuning existing harnesses to be less verbose.
- Auditing popular tools and reducing their output verbosity.
- Encouraging developers to break tasks into smaller, narrower units of work to limit context scope.
Prompt caching also plays a meaningful role at large context sizes. Both proprietary and open source LLMs support prompt caching settings. Cache writes cost money, but cached reads can dramatically reduce per-inference cost. The right cache configuration depends on your specific workload, so hand-tuning defaults to improve cache hit rate can have outsized impact.
At Databricks, relatively simple tuning of harness and caching settings produced an almost 50% reduction in generated tokens and associated costs, with no observed quality degradation for developers.
️ The AI gateway pattern
All four levers share an implicit infrastructure requirement: a central location where the model menu is managed, spend is tracked across tools, context behavior can be observed and configured, and session traces are logged for downstream analysis. This is what an AI gateway provides.
An AI gateway handles capacity management and proxying for both proprietary and open source models, budget tracking with progressive friction policies, configuration management for end-user tools, and logging of coding session traces. Databricks relies on Unity AI Gateway for these capabilities and has made it generally available. Their meta-harness, Omnigent, is also open sourced for developer tooling.
When this works and when it does not
This playbook is validated by large digital-native companies with dedicated platform engineering teams. The infrastructure components (AI gateway, meta-harness, automated internal evaluations) require upfront investment to configure and maintain. For a solo developer or a two-person team, the immediate priority is simpler: turn on spend visibility, resist locking into a single model family, and run a quick internal evaluation before adopting the next hyped model release.
The core insight transfers regardless of team size: the efficiency frontier advances faster than the intelligence frontier, and the biggest wins come from staying close to it rather than defaulting to the most capable model for every request.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
