Vibe coding is functionally fine. Securely? Only 11.8% pass.
Vibe coding gets software shipped fast. Whether that software is safe to run in production is a different question, and a new benchmark paper has a troubling answer.
What the Research Found
Researchers from LeiLiLab built SUSVIBES, a benchmark of 186 feature-request software engineering tasks drawn from real-world open-source projects. Each task was one where a human programmer had already committed a vulnerable implementation, making it a realistic proxy for production code risk.
They ran 12 agentic coding setups against the benchmark using frontier models. Every single agent performed poorly on security. The best result came from SWE-Agent with Claude Sonnet 4: 57% of its solutions were functionally correct, but only 11.8% were secure.
The Fix Attempts Did Not Work
The researchers also tested a basic mitigation: adding vulnerability hints directly to the feature request prompt. The idea was that if the agent knew what class of vulnerability to watch for, it might avoid it. According to the paper, this approach did not resolve the security issues.
The Operator Takeaway
If you are shipping features with AI-generated code into a production environment, especially anything handling authentication, file access, or user input, the benchmark suggests you should treat that code as untrusted by default. Functional tests passing is not a signal that the code is safe. Security review is a separate step and, according to this data, the one vibe coding currently skips most often.
The dataset and code are available at github.com/LeiLiLab/susvibes. The leaderboard is at leililab.github.io/susvibes-leaderboard.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
