How AI coding agents refactor Java code: a study of 51,000+ instances
A team of six researchers just published the largest empirical study to date on how AI coding agents handle refactoring in real-world open-source Java projects. The dataset covers 51,412 refactoring instances across 12,256 pull requests and 14,998 commits, drawn from the AIDev dataset. The tools studied include OpenAI Codex, Claude Code, and Cursor.
What the numbers show
Refactoring is not incidental in agentic workflows. Agents explicitly targeted refactoring in 26.1% of commits. The deeper analysis focused on the 3,907 commits (32,611 instances) where refactoring intent was stated clearly.
High-level refactorings make up the largest category overall. But agents lean slightly more toward low-level, consistency-oriented edits than human developers do: 27.6% vs. 24.4%. The two most common operations were Rename Variable (7.6%) and Change Variable Type (6.8%).
Why agents say they refactor
The stated motivations skew heavily toward internal quality. Maintainability accounted for 52.5% of reasons. Readability came in at 28.1%. External behavior was not the point.
Do the metrics back that up?
Partially. Code quality metrics showed small but statistically significant improvements in structural measures, particularly for medium-level changes. Class size and complexity both decreased. For the frequent low-level rename-and-retype operations, the measured metric improvements were limited.
The gap between stated motivation and measured outcome is worth watching. Agents are doing a lot of renaming in the name of maintainability, but the measurable structural payoff is concentrated in mid-level changes, not the small stuff.
The full paper is available on arXiv.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
