AT&T cut AI coding costs 56% by routing cheap queries to open-source models
AT&T is processing roughly 45 billion tokens a day through its internal AI platform, Ask AT&T. At that scale, model selection is a billing line item, not an academic debate. The company just published what a disciplined routing strategy actually delivers: a 56% reduction in AI coding costs with only a 2% drop in output quality.
How the routing works
AT&T implemented model routing through LiteLLM, which acts as a traffic cop between incoming employee queries and the model tier best suited to answer them. Routine and coding-related questions get redirected to lower-cost open-source models. Complex tasks still go to premium models from OpenAI and Anthropic.
AT&T VP Mark Austin noted that open-source models now trail their proprietary counterparts by roughly 6 to 10 months in capability, a gap that keeps shrinking. Open models currently handle about 40% of employee queries. The company is targeting 60 to 70% in the near term. The open-source models in active use include Nvidia’s Nemotron, Meta’s Llama, and Google’s Gemma, with additional alternatives under evaluation.
Domain-tuned models pushed savings further
Between February and July 2026, AT&T ran experiments with custom telecom-tuned models built for industry-specific queries: network troubleshooting, customer service scripts, internal documentation lookups. Those models delivered up to 90% savings in inference costs at scale compared to general-purpose alternatives.
The operator takeaway
You are not AT&T, but the underlying logic scales down. If you are running any AI workflow with meaningful volume, the most practical cost lever is not negotiating API pricing. It is classifying your queries by complexity and routing cheap ones to cheaper models. Goldman Sachs flagged AT&T’s task-specific routing approach as a potential template for enterprise AI cost management more broadly. The pattern is simple enough that any operator running multi-step AI pipelines can apply it today.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
