DeepSWE: The Coding Agent Benchmark That Fixes SWE-bench's Blind Spots

DeepSWE is a software engineering benchmark released by Datacurve AI in May 2026, designed to address contamination and specificity problems in existing benchmarks like SWE-bench. It evaluates coding agents on 113 original, long-horizon engineering tasks drawn from 91 active open-source repositories.

What's wrong with SWE-bench

SWE-bench has been the dominant coding agent benchmark since 2024, but it has two well-documented problems:

• Contamination — tasks are sourced from existing GitHub issues and pull requests. Models trained on web data have likely seen the solutions, inflating scores
• Over-specification — public benchmarks sourced from GitHub issues often carry more detail: reproduction steps, additional context, code snippets, and tests that assume specific symbols or signatures. Real engineering tasks don't come pre-packaged this way

How DeepSWE works differently

• Original tasks only — every task is written from scratch rather than adapted from existing commits, eliminating contamination risk
• Behavior-focused prompts — DeepSWE prompts are aligned with the way developers talk to their agents: behavior-focused, short, and free of large interface-definition blocks. Agents must discover where and how to implement the change themselves
• Long-horizon scope — tasks require end-to-end exploration across real repositories, not just isolated function completion
• 5 languages — TypeScript, Go, Python, JavaScript, and Rust across 91 repositories
• Program-based verifiers — automated test-based verification, not LLM-as-judge scoring

DeepSWE leaderboard (as of June 2026)

GPT-5.5's 67% at $7.23 per task makes OpenAI the clear value leader on DeepSWE. The gap between Opus 4.8 (59%) and Sonnet 4.6 (30%) is large — suggesting long-horizon engineering still strongly favors larger models. DeepSeek V4 and Mistral have not been evaluated on DeepSWE v1.1 as of June 2026, leaving a gap on the leaderboard.

Why these scores look lower than SWE-bench

Most models score 20-40 percentage points lower on DeepSWE than on SWE-bench Verified. This gap is the benchmark's point — it removes the contamination and over-specification that inflate SWE-bench scores. DeepSWE scores reflect what agents can actually do on novel, realistic engineering tasks.

DeepSWE vs other coding benchmarks