Vibe coding produces working code, but only 11.8% is secure
Your AI coding agent can ship a feature that passes every test and still leave a security hole a junior dev would catch. A new benchmark called SUSVIBES just put numbers on exactly how wide that gap is.
What the Benchmark Found
Researcher Danqing Wang built SUSVIBES from 186 real-world software engineering tasks pulled from open-source projects. Each task maps to a case where a human programmer had already committed vulnerable code while building that exact feature. The idea: see whether AI agents make the same mistake humans already documented.
Twelve agentic coding setups running frontier models were tested. Every single one scored poorly on security. The SWE-Agent framework paired with Claude Sonnet 4 produced the sharpest illustration of the gap: 57% of solutions were functionally correct, but only 11.8% of those were actually secure. That means more than four out of five working solutions still carried a known vulnerability class.
The Hint Strategy Did Not Work
The researchers tried a straightforward mitigation: add explicit vulnerability hints to the prompt, essentially telling the agent what kind of flaw to watch for. It did not meaningfully improve security outcomes. The models appear to lack a reliable internal check for security risk even when pointed directly at the problem.
Why This Matters for Operators
Code that runs and passes a functional review looks production-ready. The vulnerability only shows up in a security audit or, worse, after an incident. For any team using vibe coding in financial systems, infrastructure, or anything with real attack surface, that 11.8% secure rate is the number to benchmark against before you reduce human review time.
The SUSVIBES code, dataset, and a public leaderboard tracking agent configurations are available on GitHub.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
