SWT-bench Verified: writing tests that reproduce real GitHub issues (SWT-bench Verified)
We show this table for reference; we do not rank on it.
A test generation benchmark from LogicStar AI and ETH Zurich's SRI Lab. For each of 433 real GitHub issues from SWE-bench Verified, an agent writes a test that reproduces the reported bug: the test has to fail on the original code and pass once the issue's fix is applied. The score is the share of issues reproduced this way.
Success rate on SWT-bench Verified — November 20, 2026 snapshot
We mirror the published success rate view for SWT-bench Verified. Bugstack / Claude Sonnet 5.5 + Opus 5.5 leads the public snapshot at 91.5%, followed by Claude Code / Claude Opus 5.5 (87.8%) and DPIAgent / GPT-5 (with test selection) (87.5%). We do not use these results to rank models overall.
Bugstack / Claude Sonnet 5.5 + Opus 5.5
Unit test mode · coverage increase 72.2% · listed 2026-11-02
Claude Code / Claude Opus 5.5
Anthropic
Unit test mode · coverage increase 70.0% · listed 2026-11-03
DPIAgent / GPT-5 (with test selection)
Unit test mode · coverage increase 71.6% · listed 2026-10-23
21 modelsCodingCurrentDisplay onlyUpdated November 20, 2026 snapshot
Success rate table (21 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Bugstack / Claude Sonnet 5.5 + Opus 5.5Creator not reportedUnit test mode · coverage increase 72.2% · listed 2026-11-02 | 91.5% | Not reported | Unknown |
| 2 | Claude Code / Claude Opus 5.5AnthropicUnit test mode · coverage increase 70.0% · listed 2026-11-03 | 87.8% | Not reported | Closed |
| 3 | DPIAgent / GPT-5 (with test selection)Creator not reportedUnit test mode · coverage increase 71.6% · listed 2026-10-23 | 87.5% | Not reported | Unknown |
| 4 | LogicStar AI / L*Agent v1Creator not reportedUnit test mode · coverage increase 67.7% · listed 2025-09-13 | 84.0% | Not reported | Unknown |
| 5 | PatchTwin+Guard / GPT-5Creator not reportedUnit test mode · coverage increase 78.7% · listed 2026-07-08 | 82.9% | Not reported | Unknown |
| 6 | DPIAgent / GPT-5Creator not reportedUnit test mode · coverage increase 68.5% · listed 2026-10-23 | 81.5% | Not reported | Unknown |
| 7 | OpenHands / GPT-5Creator not reportedUnit test mode · coverage increase 66.3% · listed 2025-08-22 | 79.8% | Not reported | Unknown |
| 8 | DPIAgent / Claude Opus 4.5AnthropicUnit test mode · coverage increase 65.6% · listed 2026-10-23 | 78.8% | Not reported | Closed |
| 9 | DPIAgent / GPT-5 MiniOpenAIUnit test mode · coverage increase 71.3% · listed 2026-10-23 | 78.1% | Not reported | Closed |
| 10 | Claude Code / Claude Sonnet 5.5AnthropicUnit test mode · coverage increase 61.2% · listed 2026-11-03 | 75.8% | Not reported | Closed |
| 11 | PatchTwin+Guard / GPT-5-miniOpenAIUnit test mode · coverage increase 77.7% · listed 2026-07-08 | 75.8% | Not reported | Closed |
| 12 | ReProAgent / GPT-5-miniOpenAIUnit test mode · coverage increase 64.7% · listed 2026-04-20 | 69.7% | Not reported | Closed |
| 13 | OpenHands / GPT-5-miniOpenAIUnit test mode · coverage increase 60.6% · listed 2025-08-22 | 62.4% | Not reported | Closed |
| 14 | e-Otter++ / Claude 3.7 SonnetAnthropicUnit test mode · coverage increase 62.3% · listed 2025-08-11 | 62.1% | Not reported | Closed |
| 15 | Amazon Q Developer Agent / v20250405-devCreator not reportedUnit test mode · coverage increase 57.4% · listed 2025-04-10 | 51.0% | Not reported | Unknown |
| 16 | AssertFlip / GPT-4oOpenAIUnit test mode · coverage increase 47.4% · listed 2025-07-28 | 45.5% | Not reported | Closed |
| 17 | Otter++ / GPT-4oOpenAIUnit test mode · coverage increase 42.8% · listed 2025-03-10 | 37.4% | Not reported | Closed |
| 18 | Otter / GPT-4oOpenAIUnit test mode · coverage increase 37.6% · listed 2025-03-10 | 31.6% | Not reported | Closed |
| 19 | OpenHands / Cl. Sonnet 3.5AnthropicUnit test mode · coverage increase 52.9% · listed 2025-02-28 | 27.7% | Not reported | Closed |
| 20 | LIBRO / GPT-4oOpenAIUnit test mode · coverage increase 38.0% · listed 2025-02-28 | 17.8% | Not reported | Closed |
| 21 | Zero-Shot Plus / GPT-4o + BM25OpenAIUnit test mode · coverage increase 34.0% · listed 2025-02-28 | 14.3% | Not reported | Closed |
SWT-bench Lite
BenchLM mirrors the success rate from the SWT-bench Lite table, where the maintainers re-run each submission themselves. SWT-bench is excluded from BenchLM overall and category rankings.
| Source rank | Source model or configuration | Success rate | Source setup | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | DevstralTestGen / Devstral 2 | 89.1% | Reproduction script mode · coverage increase 57.6% · listed 2026-03-13Original result | Not reported | Unknown |
| 1 | ReProAgent / GPT-5-miniOpenAI | 56.2% | Unit test mode · coverage increase 64.8% · listed 2026-04-20Original result | Not reported | Closed |
| 2 | e-Otter++ / Claude 3.7 SonnetAnthropic | 52.5% | Unit test mode · coverage increase 56.4% · listed 2025-08-11Original result | Not reported | Closed |
| 2 | AEGIS | 47.8% | Reproduction script mode · coverage increase 26.0% · listed 2025-02-17Original result | Not reported | Unknown |
| 3 | Amazon Q Developer Agent / v20250405-dev | 39.9% | Unit test mode · coverage increase 52.7% · listed 2025-04-10Original result | Not reported | Unknown |
| 4 | AssertFlip / GPT-4oOpenAI | 38.0% | Unit test mode · coverage increase 44.2% · listed 2025-07-28Original result | Not reported | Closed |
| 5 | OpenHands / Cl. Sonnet 3.5, CI setupAnthropic | 28.3% | Unit test mode · coverage increase 52.4% · listed 2025-02-18Original result | Not reported | Closed |
| 6 | OpenHands / Cl. Sonnet 3.5, vanillaAnthropic | 22.8% | Unit test mode · coverage increase 43.6% · listed 2025-02-18Original result | Not reported | Closed |
| 7 | SWE-Agent+ / GPT-4 | 18.5% | Unit test mode · coverage increase 27.6% · listed 2024-05-22Original result | Not reported | Unknown |
| 8 | SWE-Agent / Mistral Large 2Mistral | 16.3% | Unit test mode · coverage increase 23.0% · listed 2024-05-22Original result | Not reported | Closed |
| 9 | SWE-Agent / GPT-4 | 15.9% | Unit test mode · coverage increase 26.5% · listed 2024-05-22Original result | Not reported | Unknown |
| 10 | LIBRO / GPT-4 | 14.1% | Unit test mode · coverage increase 23.8% · listed 2024-05-22Original result | Not reported | Unknown |
| 11 | Aider / GPT-4 | 12.7% | Unit test mode · coverage increase 27.8% · listed 2024-05-22Original result | Not reported | Unknown |
| 12 | SWE-Agent / Cl. 3.5 SonnetAnthropic | 12.3% | Unit test mode · coverage increase 30.3% · listed 2024-05-22Original result | Not reported | Closed |
| 13 | SWE-Agent / GPT-4o miniOpenAI | 9.8% | Unit test mode · coverage increase 20.9% · listed 2024-05-22Original result | Not reported | Closed |
| 14 | Zero-Shot Plus / GPT-4 + BM25 | 9.4% | Unit test mode · coverage increase 21.5% · listed 2024-05-22Original result | Not reported | Unknown |
| 15 | AutoCodeRover / GPT-4 | 9.1% | Unit test mode · coverage increase 17.9% · listed 2024-05-22Original result | Not reported | Unknown |
| 16 | Zero-Shot Base / GPT-4 + BM25 | 3.6% | Unit test mode · coverage increase 7.6% · listed 2024-05-22Original result | Not reported | Unknown |
| 17 | SWE-Agent / Claude 3 HaikuAnthropic | 2.5% | Unit test mode · coverage increase 3.0% · listed 2024-05-22Original result | Not reported | Closed |
| 18 | SWE-Agent / Mixtral 8x22BMistral | 0.7% | Unit test mode · coverage increase 0.9% · listed 2024-05-22Original result | Not reported | Open |
How to read this leaderboard
Higher is better. Bugstack leads Verified at 91.5%, with Claude Sonnet 5.5 writing tests and Claude Opus 5.5 choosing among them, ahead of Claude Code with Claude Opus 5.5 (87.8%) and Microsoft's DPIAgent with GPT-5 and test selection (87.5%). The agent around a model moves the score a lot: GPT-5 mini reaches 78.1% in DPIAgent, 75.8% in PatchTwin+Guard and 62.4% in OpenHands. Coverage does not follow success rate either: PatchTwin+Guard with GPT-5 covers the most of the fix (78.7%) while placing fifth among unit test entries on success rate.
Operator receipt: 21 sourced rows are currently displayable on this page; the leading published row is Bugstack / Claude Sonnet 5.5 + Opus 5.5 at 91.5%.
Honest limit: Each row rates an agent system and its model together, often with extra steps such as generating several candidate tests and selecting one, so it is not a model-only score. Entries are dated by submission, from May 2024 to November 2026. The maintainers' September 13, 2025 release fixed grading issues and re-scored many entries, mostly upward by 2 to 3 points; rows it did not re-score, including the 2024 paper baselines, keep their original numbers. The issues come from public Python repositories that models may have seen in training. BenchLM shows SWT-bench Verified as display only; it does not enter overall or category rankings.
How SWT-bench Verified is shown here
We mirror the SWT-bench Verified table from swtbench.com. Each of the 433 tasks is a real GitHub issue from SWE-bench Verified. The agent reads the issue and the repository before the fix and adds tests; an issue counts as reproduced when the tests fail on the original code and pass after the developer's fix. The score is the share of issues reproduced, and coverage increase is how much of the fix's changed code the tests run.
Each row is an agent system with the model it ran, as the maintainers list it, so it rates the two together. SWT-bench is display only on BenchLM. The maintainers rank reproduction script mode, a stand-alone script outside the project's test suite, separately, so it is not in this table: TEX-T / Claude 4 Sonnet reproduces 87.0% (rank 1 in that mode).
Snapshot
The published SWT-bench Verified snapshot places Bugstack / Claude Sonnet 5.5 + Opus 5.5 first at 91.5%. The third row is 4.0 points behind. The broader top-10 range is 15.7 points, so the table still separates the published systems.
21 models have been evaluated on SWT-bench Verified. The benchmark falls in the Coding category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. SWT-bench Verified is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About SWT-bench Verified
Year
2024
Tasks
433 human-verified GitHub issues from SWE-bench Verified (Python repositories); SWT-bench Lite has 276 issues from SWE-bench Lite
Format
Agent or model gets the issue and the pre-fix repository and adds tests; tests run before and after the reference fix
Difficulty
Reproducing a bug from an issue report as a failing test inside a real project's test suite
SWT-bench uses the same Python repositories and issues as SWE-bench, but asks for the test instead of the fix. The agent gets the issue text and the repository as it was before the fix, and adds tests. SWT-bench then runs them on the original code, where they must fail, and again after applying the developer's own fix, where they must pass. An issue counts as reproduced only when both happen. The second figure, coverage increase, is the mean line coverage the generated tests reach on the lines the fix changes. In unit test mode the tests go into the project's own test suite; reproduction script mode, marked separately, accepts a stand-alone script that skips the test framework. Verified is the 433 human-verified solvable issues of SWE-bench Verified, and Lite is 276 issues from SWE-bench Lite. The maintainers run every submitted set of predictions themselves before listing it, and link each entry's trajectories.
Freshness and provenance
Version
SWT-bench Verified (433 issues), live leaderboard
Refresh cadence
Rolling leaderboard
Staleness state
Current
Question availability
Public: datasets on Hugging Face and the evaluation harness on GitHub (MIT); entries link their trajectories
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does SWT-bench measure?
Whether an AI agent can turn a bug report into a test that proves the bug exists. For each real GitHub issue, the generated test must fail on the original code and pass after the developer's fix. SWE-bench asks for the fix; SWT-bench asks for the test.
Which agent leads SWT-bench Verified?
Bugstack, at 91.5% with Claude Sonnet 5.5 and Claude Opus 5.5, followed by Claude Code with Claude Opus 5.5 at 87.8% and DPIAgent with GPT-5 and test selection at 87.5% (leaderboard of November 4, 2026).
What is the difference between unit test mode and script mode on SWT-bench?
In unit test mode the agent adds tests to the project's own test suite. In reproduction script mode it writes a stand-alone script that reproduces the issue without using the test framework. The leaderboard marks script mode entries and awards its medals separately for each mode.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.