Why your AI coding ROI metrics are lying to your CFO
Your board spent millions on AI coding tools. Now they want the number. And most engineering leaders cannot produce one that holds up under scrutiny.
The usual answer is Jira tickets, story points, or adoption percentages. Those metrics tell you what teams planned, how big they thought a task was, and who opened a tool this week. None of them tell you what actually shipped. Under agentic coding, the gap between those proxies and reality gets wider every sprint.
Ted Julian, Founder and CEO of Flux, laid out the problem and a code-first solution in a recent episode of The Agentic Enterprise on TFiR. Here is what engineering leaders, and the operators who build on top of engineering teams, need to understand.
️ Why Legacy Metrics Break Under Agentic Coding
Julian’s framing is direct: existing measurement tracks derivatives of work, not work itself. Jira tickets and GitHub tagging were imperfect before agentic coding arrived. They become actively misleading once code generation accelerates output beyond what those systems were built to track.
“The tools that we have used in the past are no longer appropriate for the job at hand.” — Ted Julian, Founder and CEO, Flux
The board pressure compounds the problem. The typical sequence: 18 months ago, leadership demanded teams go all in on AI. Six months later, the same board asked whether millions of dollars in tokens were actually working. Engineering leaders caught in that whiplash had no credible answer because the measurement infrastructure was never built for this question.
Teams often try to roll their own dashboards with a coding assistant. Julian says that spiral is predictable: six dashboards become twelve, analysis piles on top of analysis, and eventually a VP of engineering realizes they are building another application instead of running their core business.
The Rework Blind Spot
Rework is the clearest example of how vanity metrics mislead. A team shows rising story-point velocity. Leadership socializes the number. But some of that code never made it to production on the timeline the metric implied. It was cleaned up after testing caught a problem, after a performance analysis flagged an issue, or after an unconsidered use case caused failures.
Every bit of rework eats into the supposed ROI of that velocity number. Julian’s point is that rework is common with generated code and it could erase the entire apparent gain.
“If you’re blind on this rework, then you have a major blind spot in terms of understanding the impact of CodeGen in your AI transformation.” — Ted Julian, Founder and CEO, Flux
Rework is one problem. Review debt is another.
⏱️ Review Debt: Act 2 of Every AI Transformation
Julian calls this act 2. In act 1, a few teams adopt code generation and post impressive results. Everyone is excited. Three to six months later, those same teams see lead time for change start to grow. The reason is review debt: AI generated ten times more code, but review capacity did not scale by a factor of ten.
“You’re generating 10 times as much code. Do you have 10 times as much code review capacity? Probably not.” — Ted Julian, Founder and CEO, Flux
Even when teams use AI to assist with review, humans stay in the loop. That is the right call, Julian says, because things that slip will cost you. The real trap is not having visibility into where review debt is accumulating and why.
There is a people dimension here that Julian says gets almost no attention. Senior developers often absorb review debt across multiple teams, not just their own. If your team definitions in GitHub are static and top-down, that senior developer looks like a productivity drop. Her metrics seem to show she is contributing less to her assigned team. The truth is she is contributing across several teams in ways the tooling cannot see. Leaders without accurate team visibility could make a completely wrong decision about that person.
The Code-First Approach
Julian’s answer is to treat the code itself as the only source of truth. Flux uses AI and static analysis to build an understanding of where the codebase stands, how it is changing sprint over sprint, and what that means for velocity and quality.
“The code really is the only source of truth. It’s the only thing you can truly trust when trying to answer these questions.” — Ted Julian, Founder and CEO, Flux
Because the analysis runs directly on the code, it requires no integrations with GitHub, Jira, or ticketing systems. Julian notes that large organizations often have three Jira instances configured differently from each other. Reconciling them adds onboarding toil that never fully goes away. A code-first approach sidesteps that entirely and produces a consistent view across the whole organization from day one.
Teams give Flux access to their code and get results within an hour or two.
The Board Slide That Proved ROI: 9% to 43% Feature Work
The most concrete customer story Julian shares involves a new VP of engineering who started in June of last year. When onboarding with Flux, the team backfilled all pull requests to January of that year.
Flux classifies each pull request by work type using an LLM: feature work, maintenance, bug fix, or a weighted split when a PR covers more than one type. By December, the data showed feature work had grown from 9% of engineering effort at the start of the year to 43% by year end.
That became a board slide. Board members asked why they were not seeing this data from all their engineering teams. The result landed well beyond optics: finance saw the slide before the meeting and reached out to say they use feature work as a proxy for R&D in the software organization, and that proxy drives R&D tax credit claims in the US. The jump from 9% to 43% supported a significant increase in those credits. The CFO then told the VP to claim the tools used to generate that increase as an internal write-off.
“It’s going to be more auditable because you have the receipts, every pull request.” — Ted Julian, Founder and CEO, Flux
Julian contrasts that with how most organizations handle R&D tax credit calculations today: a few developers eyeball a month of pull requests, estimate a percentage, and fill in a shared spreadsheet. A pull-request-based classification is more defensible under audit because the evidence is explicit and granular.
Sprint Digests: From 45-Minute Sundays to 5 Minutes
Flux is rolling out biweekly digests, currently being tested with early customers. The digest arrives at the end of each two-week sprint and contains three things:
- An AI summary of what the team actually did, derived from the code rather than ticket descriptions
- DORA metrics for those two weeks
- Items to watch: new library dependencies that may carry security issues, downward trends in DORA metrics with an agentic root-cause suggestion
On the root-cause side, Julian gives an example: lead time for change slows, Flux analyzes recent comments, and surfaces a likely cause: the team is iterating back and forth trying to patch a vulnerability without breaking the application.
“It’s taking that Sunday activity from 45 minutes plus to a 5-minute exercise.” — Ted Julian, Founder and CEO, Flux
Early customers report cutting the Sunday review from 45 minutes or more down to about 5 minutes and arriving Monday with a clearer picture of who to talk to and what to discuss.
Background: Why Flux Exists
Flux was founded about four years ago and pivoted roughly two years in to focus on using AI to understand code as a way to better manage code creation and the teams behind it. The pivot timing lined up with GitHub Copilot kicking off the code generation wave. Julian says the team anticipated the shift would strain existing management approaches and create demand for something better suited to the new environment.
The Verdict
If you are running an engineering team, approving AI tooling budgets, or advising a founder who is, the core argument here is worth taking seriously: story points and Jira tickets were imperfect proxies before agentic coding and they are worse now. A pull-request-based approach gives you auditable evidence, surfaces rework and review debt before they compound, and produces the kind of board-ready output that can translate directly into R&D tax credit claims.
The Flux approach is described at askflux.ai. The full interview is available on YouTube.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
