AI coding agents fully solve just 26% of real tasks, new benchmark finds
On the benchmarks most AI labs cite, coding agents now score above 99 percent. A new test from Alibaba researchers suggests that number tells you almost nothing about how these tools perform in practice.
What Co-Design Bench Measures
The benchmark, called Co-Design Bench, pulls from real coding sessions rather than synthetic problems. The researchers specifically selected tasks where users had to rephrase requests multiple times or roll back changes, meaning the test is weighted toward cases that already gave AI agents trouble. Tasks range from small edits, like pulling a dollar amount from a receipt and updating every dependent total in a web app, to full project overhauls.
Grading is strict. One AI judge tests the finished app in a browser. A second re-checks every positive verdict and looks for mock-ups or hardcoded results. Points go only to features that actually work in the running application.
The Numbers
GPT-6 Astra led all ten models tested. It meets about 70 percent of a task’s requirements on average. But it fully completes only 26.4 percent of tasks. In most cases the agent handles the bulk of the work and leaves at least one detail unfinished, so a human still has to clean up.
- GPT-6 Astra: 70% of requirements met on average, 26.4% of tasks fully correct
- Claude Opus 5 and Qwen3.8-Max took the next two spots
- Grok 4.6 landed last among the ten models tested
- On larger overhauls, the smaller Qwen3.8-27B climbed to second place
Performance dropped across every model once a task combined multiple change types, including layout, design, interaction, and content. That combination is the norm in real projects, not the exception.
Throwing More Compute at It Does Not Help
GLM-5.2 burned through nearly six times as many tokens as GPT-6 Astra on smaller tasks and still scored significantly worse. Extra compute did not compensate for the underlying gap.
The Harness Problem
The environment an agent runs in, the program through which it reads, writes, and executes code, shaped results as much as the model itself. Qwen3.8-27B scored either 75 or 56 percent on larger overhauls depending solely on which harness was used. Which of two models came out ahead also flipped depending on the environment.
The authors recommend always reporting model and working environment together. For operators picking a coding agent, that means testing both components as a pair, not trusting a leaderboard that lists only the model.
Context on Benchmark Reliability
Co-Design Bench has limits of its own. It evaluates one fixed request per task and does not account for follow-up questions or mid-task corrections from users. The AI judges can also make mistakes.
A separate study by the research organization METR found that the actual acceptance rate of AI-generated code by project leads was roughly 24 percentage points below scores on the widely used SWE-bench Verified. Flaws in test environments can also let agents pass tasks without actually solving them.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
