Skip to main content
BenchLM
Data

SWT-bench Verified: writing tests that reproduce real GitHub issues (SWT-bench Verified)

We show this table for reference; we do not rank on it.

A test generation benchmark from LogicStar AI and ETH Zurich's SRI Lab. For each of 433 real GitHub issues from SWE-bench Verified, an agent writes a test that reproduces the reported bug: the test has to fail on the original code and pass once the issue's fix is applied. The score is the share of issues reproduced this way.

Success rate on SWT-bench Verified — November 20, 2026 snapshot

We mirror the published success rate view for SWT-bench Verified. Bugstack / Claude Sonnet 5.5 + Opus 5.5 leads the public snapshot at 91.5%, followed by Claude Code / Claude Opus 5.5 (87.8%) and DPIAgent / GPT-5 (with test selection) (87.5%). We do not use these results to rank models overall.

21 modelsCodingCurrentDisplay onlyUpdated November 20, 2026 snapshot

Success rate table (21 models)

Success rate results for SWT-bench Verified
RankModel / configurationScoreParameters (B)Open / closed
1Bugstack / Claude Sonnet 5.5 + Opus 5.5Creator not reportedUnit test mode · coverage increase 72.2% · listed 2026-11-02
91.5%
Not reportedUnknown
2Claude Code / Claude Opus 5.5AnthropicUnit test mode · coverage increase 70.0% · listed 2026-11-03
87.8%
Not reportedClosed
3DPIAgent / GPT-5 (with test selection)Creator not reportedUnit test mode · coverage increase 71.6% · listed 2026-10-23
87.5%
Not reportedUnknown
4LogicStar AI / L*Agent v1Creator not reportedUnit test mode · coverage increase 67.7% · listed 2025-09-13
84.0%
Not reportedUnknown
5PatchTwin+Guard / GPT-5Creator not reportedUnit test mode · coverage increase 78.7% · listed 2026-07-08
82.9%
Not reportedUnknown
6DPIAgent / GPT-5Creator not reportedUnit test mode · coverage increase 68.5% · listed 2026-10-23
81.5%
Not reportedUnknown
7OpenHands / GPT-5Creator not reportedUnit test mode · coverage increase 66.3% · listed 2025-08-22
79.8%
Not reportedUnknown
8DPIAgent / Claude Opus 4.5AnthropicUnit test mode · coverage increase 65.6% · listed 2026-10-23
78.8%
Not reportedClosed
9DPIAgent / GPT-5 MiniOpenAIUnit test mode · coverage increase 71.3% · listed 2026-10-23
78.1%
Not reportedClosed
10Claude Code / Claude Sonnet 5.5AnthropicUnit test mode · coverage increase 61.2% · listed 2026-11-03
75.8%
Not reportedClosed
11PatchTwin+Guard / GPT-5-miniOpenAIUnit test mode · coverage increase 77.7% · listed 2026-07-08
75.8%
Not reportedClosed
12ReProAgent / GPT-5-miniOpenAIUnit test mode · coverage increase 64.7% · listed 2026-04-20
69.7%
Not reportedClosed
13OpenHands / GPT-5-miniOpenAIUnit test mode · coverage increase 60.6% · listed 2025-08-22
62.4%
Not reportedClosed
14e-Otter++ / Claude 3.7 SonnetAnthropicUnit test mode · coverage increase 62.3% · listed 2025-08-11
62.1%
Not reportedClosed
15Amazon Q Developer Agent / v20250405-devCreator not reportedUnit test mode · coverage increase 57.4% · listed 2025-04-10
51.0%
Not reportedUnknown
16AssertFlip / GPT-4oOpenAIUnit test mode · coverage increase 47.4% · listed 2025-07-28
45.5%
Not reportedClosed
17Otter++ / GPT-4oOpenAIUnit test mode · coverage increase 42.8% · listed 2025-03-10
37.4%
Not reportedClosed
18Otter / GPT-4oOpenAIUnit test mode · coverage increase 37.6% · listed 2025-03-10
31.6%
Not reportedClosed
19OpenHands / Cl. Sonnet 3.5AnthropicUnit test mode · coverage increase 52.9% · listed 2025-02-28
27.7%
Not reportedClosed
20LIBRO / GPT-4oOpenAIUnit test mode · coverage increase 38.0% · listed 2025-02-28
17.8%
Not reportedClosed
21Zero-Shot Plus / GPT-4o + BM25OpenAIUnit test mode · coverage increase 34.0% · listed 2025-02-28
14.3%
Not reportedClosed

SWT-bench Lite

BenchLM mirrors the success rate from the SWT-bench Lite table, where the maintainers re-run each submission themselves. SWT-bench is excluded from BenchLM overall and category rankings.

SWT-bench Lite; source configurations remain separate from the canonical benchmark table.
Source rankSource model or configurationSuccess rateSource setupParameters (B)Open / closed
1DevstralTestGen / Devstral 289.1%Reproduction script mode · coverage increase 57.6% · listed 2026-03-13Original resultNot reportedUnknown
1ReProAgent / GPT-5-miniOpenAI56.2%Unit test mode · coverage increase 64.8% · listed 2026-04-20Original resultNot reportedClosed
2e-Otter++ / Claude 3.7 SonnetAnthropic52.5%Unit test mode · coverage increase 56.4% · listed 2025-08-11Original resultNot reportedClosed
2AEGIS47.8%Reproduction script mode · coverage increase 26.0% · listed 2025-02-17Original resultNot reportedUnknown
3Amazon Q Developer Agent / v20250405-dev39.9%Unit test mode · coverage increase 52.7% · listed 2025-04-10Original resultNot reportedUnknown
4AssertFlip / GPT-4oOpenAI38.0%Unit test mode · coverage increase 44.2% · listed 2025-07-28Original resultNot reportedClosed
5OpenHands / Cl. Sonnet 3.5, CI setupAnthropic28.3%Unit test mode · coverage increase 52.4% · listed 2025-02-18Original resultNot reportedClosed
6OpenHands / Cl. Sonnet 3.5, vanillaAnthropic22.8%Unit test mode · coverage increase 43.6% · listed 2025-02-18Original resultNot reportedClosed
7SWE-Agent+ / GPT-418.5%Unit test mode · coverage increase 27.6% · listed 2024-05-22Original resultNot reportedUnknown
8SWE-Agent / Mistral Large 2Mistral16.3%Unit test mode · coverage increase 23.0% · listed 2024-05-22Original resultNot reportedClosed
9SWE-Agent / GPT-415.9%Unit test mode · coverage increase 26.5% · listed 2024-05-22Original resultNot reportedUnknown
10LIBRO / GPT-414.1%Unit test mode · coverage increase 23.8% · listed 2024-05-22Original resultNot reportedUnknown
11Aider / GPT-412.7%Unit test mode · coverage increase 27.8% · listed 2024-05-22Original resultNot reportedUnknown
12SWE-Agent / Cl. 3.5 SonnetAnthropic12.3%Unit test mode · coverage increase 30.3% · listed 2024-05-22Original resultNot reportedClosed
13SWE-Agent / GPT-4o miniOpenAI9.8%Unit test mode · coverage increase 20.9% · listed 2024-05-22Original resultNot reportedClosed
14Zero-Shot Plus / GPT-4 + BM259.4%Unit test mode · coverage increase 21.5% · listed 2024-05-22Original resultNot reportedUnknown
15AutoCodeRover / GPT-49.1%Unit test mode · coverage increase 17.9% · listed 2024-05-22Original resultNot reportedUnknown
16Zero-Shot Base / GPT-4 + BM253.6%Unit test mode · coverage increase 7.6% · listed 2024-05-22Original resultNot reportedUnknown
17SWE-Agent / Claude 3 HaikuAnthropic2.5%Unit test mode · coverage increase 3.0% · listed 2024-05-22Original resultNot reportedClosed
18SWE-Agent / Mixtral 8x22BMistral0.7%Unit test mode · coverage increase 0.9% · listed 2024-05-22Original resultNot reportedOpen

How to read this leaderboard

Higher is better. Bugstack leads Verified at 91.5%, with Claude Sonnet 5.5 writing tests and Claude Opus 5.5 choosing among them, ahead of Claude Code with Claude Opus 5.5 (87.8%) and Microsoft's DPIAgent with GPT-5 and test selection (87.5%). The agent around a model moves the score a lot: GPT-5 mini reaches 78.1% in DPIAgent, 75.8% in PatchTwin+Guard and 62.4% in OpenHands. Coverage does not follow success rate either: PatchTwin+Guard with GPT-5 covers the most of the fix (78.7%) while placing fifth among unit test entries on success rate.

Operator receipt: 21 sourced rows are currently displayable on this page; the leading published row is Bugstack / Claude Sonnet 5.5 + Opus 5.5 at 91.5%.

Honest limit: Each row rates an agent system and its model together, often with extra steps such as generating several candidate tests and selecting one, so it is not a model-only score. Entries are dated by submission, from May 2024 to November 2026. The maintainers' September 13, 2025 release fixed grading issues and re-scored many entries, mostly upward by 2 to 3 points; rows it did not re-score, including the 2024 paper baselines, keep their original numbers. The issues come from public Python repositories that models may have seen in training. BenchLM shows SWT-bench Verified as display only; it does not enter overall or category rankings.

How SWT-bench Verified is shown here

We mirror the SWT-bench Verified table from swtbench.com. Each of the 433 tasks is a real GitHub issue from SWE-bench Verified. The agent reads the issue and the repository before the fix and adds tests; an issue counts as reproduced when the tests fail on the original code and pass after the developer's fix. The score is the share of issues reproduced, and coverage increase is how much of the fix's changed code the tests run.

Each row is an agent system with the model it ran, as the maintainers list it, so it rates the two together. SWT-bench is display only on BenchLM. The maintainers rank reproduction script mode, a stand-alone script outside the project's test suite, separately, so it is not in this table: TEX-T / Claude 4 Sonnet reproduces 87.0% (rank 1 in that mode).

The published SWT-bench Verified snapshot places Bugstack / Claude Sonnet 5.5 + Opus 5.5 first at 91.5%. The third row is 4.0 points behind. The broader top-10 range is 15.7 points, so the table still separates the published systems.

21 models have been evaluated on SWT-bench Verified. The benchmark falls in the Coding category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. SWT-bench Verified is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About SWT-bench Verified

Year

2024

Tasks

433 human-verified GitHub issues from SWE-bench Verified (Python repositories); SWT-bench Lite has 276 issues from SWE-bench Lite

Format

Agent or model gets the issue and the pre-fix repository and adds tests; tests run before and after the reference fix

Difficulty

Reproducing a bug from an issue report as a failing test inside a real project's test suite

SWT-bench uses the same Python repositories and issues as SWE-bench, but asks for the test instead of the fix. The agent gets the issue text and the repository as it was before the fix, and adds tests. SWT-bench then runs them on the original code, where they must fail, and again after applying the developer's own fix, where they must pass. An issue counts as reproduced only when both happen. The second figure, coverage increase, is the mean line coverage the generated tests reach on the lines the fix changes. In unit test mode the tests go into the project's own test suite; reproduction script mode, marked separately, accepts a stand-alone script that skips the test framework. Verified is the 433 human-verified solvable issues of SWE-bench Verified, and Lite is 276 issues from SWE-bench Lite. The maintainers run every submitted set of predictions themselves before listing it, and link each entry's trajectories.

Freshness and provenance

Version

SWT-bench Verified (433 issues), live leaderboard

Refresh cadence

Rolling leaderboard

Staleness state

Current

Question availability

Public: datasets on Hugging Face and the evaluation harness on GitHub (MIT); entries link their trajectories

CurrentDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does SWT-bench measure?

Whether an AI agent can turn a bug report into a test that proves the bug exists. For each real GitHub issue, the generated test must fail on the original code and pass after the developer's fix. SWE-bench asks for the fix; SWT-bench asks for the test.

Which agent leads SWT-bench Verified?

Bugstack, at 91.5% with Claude Sonnet 5.5 and Claude Opus 5.5, followed by Claude Code with Claude Opus 5.5 at 87.8% and DPIAgent with GPT-5 and test selection at 87.5% (leaderboard of November 4, 2026).

What is the difference between unit test mode and script mode on SWT-bench?

In unit test mode the agent adds tests to the project's own test suite. In reproduction script mode it writes a stand-alone script that reproduces the issue without using the test framework. The leaderboard marks script mode entries and awards its medals separately for each mode.

Last updated: November 20, 2026 snapshot · mirrored from the public benchmark leaderboard

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 7,000+ subscribers.

One email each week. Unsubscribe anytime.