Vibe coding research: +26% productivity or -19%? Both are true
A research team just published a state-of-the-art review on vibe coding, the AI-assisted development practice named by Andrej Karpathy in February 2025, where developers describe intent in natural language and validate results by running the code rather than reading it.
The Productivity Numbers Look Contradictory
The study pulls together evidence across software engineering, human-computer interaction, labour economics, security research, governance, and education. The headline findings pull in opposite directions:
- Peer-reviewed field experiments report +26% more tasks per week
- Independent randomised trials measure a 19% slowdown
- Team-level telemetry shows code-review time up +441%
The researchers argue these readings are consistent once measurement method, scope, and time horizon are held constant. The paper identifies six patterns behind the dispersion, including effect-shrinkage under broader measurement, self-report diverging from independent measurement, and output volume being conflated with actual productivity.
Where AI Coding Holds Up and Where It Does Not
Early benchmarks have saturated, but task-level capability is uneven. The review finds reliable performance on code generation, weak fault detection, and documentation that is hard to audit.
Beyond productivity, the paper documents security failures in deployed applications, code-quality degradation visible in large-scale telemetry, unsettled copyright exposure, and evidence of skill atrophy in developers who rely heavily on AI assistance.
The Falsifiable Conjecture Worth Watching
The review closes with one testable prediction: the gains are real on new code and shrink or reverse on mature codebases. If that holds up under further testing, it would account for most of the disagreement in the existing record.
For solo developers and small teams evaluating how much to lean on AI coding tools, that distinction matters. Greenfield projects and mature production codebases are genuinely different contexts.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
