DeepSWE benchmark ranks AI coding agents on real engineering work
Most AI coding benchmarks are starting to look like open-book exams. Models trained on public GitHub data can effectively recall solutions they have already seen. DeepSWE is a new benchmarking platform built to close that loophole.
What Makes It Different
DeepSWE tasks are written from scratch and never merged back into public upstream repositories, so models cannot memorize them during pre-training. The platform also strips the full git history from evaluation containers, eliminating the shortcut where agents run git log to locate the gold patch directly.
The scale difference between DeepSWE and older benchmarks is significant. The average DeepSWE reference solution edits 7 files and adds 668 lines of code. SWE-bench Pro averages 5 edited files and 120 lines. SWE-bench Verified averages 1 edited file and 10 lines.
Coverage is broader too. DeepSWE spans 91 active open-source repositories across TypeScript, Go, Python, JavaScript, and Rust. SWE-bench Pro covers 11 repositories. SWE-bench Verified covers 12.
Current Leaderboard Results
All models run on a standardized, model-agnostic harness called mini-swe-agent with identical bash tools and system prompts. No vendor-specific editing primitives. The numbers below reflect the v1.1 setup unless noted.
Top tier: 70% and above
- gpt-5.6-sol [max]: 73% pass rate, $8.39 per task, 60k output tokens
- claude-fable-5 [max]: 70% pass rate, $21.63 per task, 119k output tokens
- gpt-5.6-terra [max]: 70% pass rate, $4.95 per task, 72k output tokens
High-performing tier: 60% to 69%
- gpt-5.6-luna [max]: 67% pass rate, $3.03 per task
- gpt-5.5 [xhigh]: 67% pass rate, median 46k output tokens (most token-efficient in the upper tier)
Solid performers: 50% to 59%
- claude-opus-4.8 [max]: 59% pass rate, $13.22 per task
- claude-sonnet-5 [max]: 54% pass rate, $26.40 per task, 214k output tokens
- grok-4.5 [high]: 54% pass rate, $2.42 per task
- muse-spark-1.1 [xhigh]: 53% pass rate, $2.36 per task
- gpt-5.4 [xhigh]: 52% pass rate, $5.65 per task
Lower tier
- gemini-3.5-flash [medium]: 37%
- kimi-k2.7-code: 31%
- claude-sonnet-4.6 [high]: 30%
- gemini-3.1-pro [high]: 12%
The Cost Efficiency Story
The practical takeaway is that gpt-5.6-terra [max] matches claude-fable-5 on pass rate (both at 70%) while costing $4.95 per task versus $21.63. That is a 77% cost reduction for identical benchmark performance.
The benchmark also tests models in their native commercial environments (Claude Code, Codex CLI, Gemini CLI) via an evaluation tool called Pier. According to the benchmark data, the agent harness often matters as much as the underlying model itself.
For developers choosing a coding agent or building agent tooling, DeepSWE offers the clearest apples-to-apples comparison currently available on long-horizon, real-world engineering tasks.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
