# SWT-bench Verified: writing tests that reproduce real GitHub issues (SWT-bench Verified)

> A test generation benchmark from LogicStar AI and ETH Zurich's SRI Lab. For each of 433 real GitHub issues from SWE-bench Verified, an agent writes a test that reproduces the reported bug: the test has to fail on the original code and pass once the issue's fix is applied. The score is the share of issues reproduced this way.

Canonical page: https://benchlm.ai/benchmarks/swt-bench-verified

- Category: [Coding](/coding)
- Last updated: November 20, 2026 snapshot

## About SWT-bench Verified

- Year: 2024
- Tasks: 433 human-verified GitHub issues from SWE-bench Verified (Python repositories); SWT-bench Lite has 276 issues from SWE-bench Lite
- Format: Agent or model gets the issue and the pre-fix repository and adds tests; tests run before and after the reference fix
- Difficulty: Reproducing a bug from an issue report as a failing test inside a real project's test suite
- Paper: [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://arxiv.org/abs/2406.12952)

SWT-bench uses the same Python repositories and issues as SWE-bench, but asks for the test instead of the fix. The agent gets the issue text and the repository as it was before the fix, and adds tests. SWT-bench then runs them on the original code, where they must fail, and again after applying the developer's own fix, where they must pass. An issue counts as reproduced only when both happen. The second figure, coverage increase, is the mean line coverage the generated tests reach on the lines the fix changes. In unit test mode the tests go into the project's own test suite; reproduction script mode, marked separately, accepts a stand-alone script that skips the test framework. Verified is the 433 human-verified solvable issues of SWE-bench Verified, and Lite is 276 issues from SWE-bench Lite. The maintainers run every submitted set of predictions themselves before listing it, and link each entry's trajectories.

SWT-bench Verified is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (21 models)

| Rank | Model | Configuration | Creator | Score |
|------|-------|---------------|---------|-------|
| 1 | [Bugstack / Claude Sonnet 5.5 + Opus 5.5](https://bugstack.ai) | Unit test mode · coverage increase 72.2% · listed 2026-11-02 | — | 91.5% |
| 2 | [Claude Code / Claude Opus 5.5](/models/claude-opus-5-5) | Unit test mode · coverage increase 70.0% · listed 2026-11-03 | Anthropic | 87.8% |
| 3 | [DPIAgent / GPT-5 (with test selection)](https://neverwinhao.github.io/DPIAgent/) | Unit test mode · coverage increase 71.6% · listed 2026-10-23 | — | 87.5% |
| 4 | [LogicStar AI / L*Agent v1](https://logicstar.ai/) | Unit test mode · coverage increase 67.7% · listed 2025-09-13 | — | 84.0% |
| 5 | [PatchTwin+Guard / GPT-5](https://github.com/lbaf23/PatchTwin) | Unit test mode · coverage increase 78.7% · listed 2026-07-08 | — | 82.9% |
| 6 | [DPIAgent / GPT-5](https://neverwinhao.github.io/DPIAgent/) | Unit test mode · coverage increase 68.5% · listed 2026-10-23 | — | 81.5% |
| 7 | [OpenHands / GPT-5](https://docs.all-hands.dev/) | Unit test mode · coverage increase 66.3% · listed 2025-08-22 | — | 79.8% |
| 8 | [DPIAgent / Claude Opus 4.5](/models/claude-opus-4-5) | Unit test mode · coverage increase 65.6% · listed 2026-10-23 | Anthropic | 78.8% |
| 9 | [DPIAgent / GPT-5 Mini](/models/gpt-5-mini) | Unit test mode · coverage increase 71.3% · listed 2026-10-23 | OpenAI | 78.1% |
| 10 | [Claude Code / Claude Sonnet 5.5](/models/claude-sonnet-5-5) | Unit test mode · coverage increase 61.2% · listed 2026-11-03 | Anthropic | 75.8% |
| 11 | [PatchTwin+Guard / GPT-5-mini](/models/gpt-5-mini) | Unit test mode · coverage increase 77.7% · listed 2026-07-08 | OpenAI | 75.8% |
| 12 | [ReProAgent / GPT-5-mini](/models/gpt-5-mini) | Unit test mode · coverage increase 64.7% · listed 2026-04-20 | OpenAI | 69.7% |
| 13 | [OpenHands / GPT-5-mini](/models/gpt-5-mini) | Unit test mode · coverage increase 60.6% · listed 2025-08-22 | OpenAI | 62.4% |
| 14 | [e-Otter++ / Claude 3.7 Sonnet](/models/claude-3-7-sonnet) | Unit test mode · coverage increase 62.3% · listed 2025-08-11 | Anthropic | 62.1% |
| 15 | [Amazon Q Developer Agent / v20250405-dev](https://aws.amazon.com/q/) | Unit test mode · coverage increase 57.4% · listed 2025-04-10 | — | 51.0% |
| 16 | [AssertFlip / GPT-4o](/models/gpt-4o) | Unit test mode · coverage increase 47.4% · listed 2025-07-28 | OpenAI | 45.5% |
| 17 | [Otter++ / GPT-4o](/models/gpt-4o) | Unit test mode · coverage increase 42.8% · listed 2025-03-10 | OpenAI | 37.4% |
| 18 | [Otter / GPT-4o](/models/gpt-4o) | Unit test mode · coverage increase 37.6% · listed 2025-03-10 | OpenAI | 31.6% |
| 19 | [OpenHands / Cl. Sonnet 3.5](/models/claude-3-5-sonnet) | Unit test mode · coverage increase 52.9% · listed 2025-02-28 | Anthropic | 27.7% |
| 20 | [LIBRO / GPT-4o](/models/gpt-4o) | Unit test mode · coverage increase 38.0% · listed 2025-02-28 | OpenAI | 17.8% |
| 21 | [Zero-Shot Plus / GPT-4o + BM25](/models/gpt-4o) | Unit test mode · coverage increase 34.0% · listed 2025-02-28 | OpenAI | 14.3% |

## SWT-bench Lite

Source snapshot: November 20, 2026 snapshot.

BenchLM mirrors the success rate from the SWT-bench Lite table, where the maintainers re-run each submission themselves. SWT-bench is excluded from BenchLM overall and category rankings.

- [SWT-bench leaderboard](https://swtbench.com/)
- [SWT-Bench paper (arXiv, NeurIPS 2024)](https://arxiv.org/abs/2406.12952)
- [Evaluation harness and leaderboard data (GitHub)](https://github.com/logic-star-ai/swt-bench)

| Source model | Configuration | Success rate |
|---|---|---|
| [DevstralTestGen / Devstral 2](https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512) | Reproduction script mode · coverage increase 57.6% · listed 2026-03-13 | 89.1% |
| [ReProAgent / GPT-5-mini](https://github.com/iSEngLab/ReProAgent) | Unit test mode · coverage increase 64.8% · listed 2026-04-20 | 56.2% |
| [e-Otter++ / Claude 3.7 Sonnet](https://arxiv.org/abs/2508.06365) | Unit test mode · coverage increase 56.4% · listed 2025-08-11 | 52.5% |
| [AEGIS](https://arxiv.org/pdf/2411.18015) | Reproduction script mode · coverage increase 26.0% · listed 2025-02-17 | 47.8% |
| [Amazon Q Developer Agent / v20250405-dev](https://aws.amazon.com/q/) | Unit test mode · coverage increase 52.7% · listed 2025-04-10 | 39.9% |
| [AssertFlip / GPT-4o](https://arxiv.org/abs/2507.17542) | Unit test mode · coverage increase 44.2% · listed 2025-07-28 | 38.0% |
| [OpenHands / Cl. Sonnet 3.5, CI setup](https://docs.all-hands.dev/) | Unit test mode · coverage increase 52.4% · listed 2025-02-18 | 28.3% |
| [OpenHands / Cl. Sonnet 3.5, vanilla](https://docs.all-hands.dev/) | Unit test mode · coverage increase 43.6% · listed 2025-02-18 | 22.8% |
| [SWE-Agent+ / GPT-4](https://arxiv.org/abs/2406.12952) | Unit test mode · coverage increase 27.6% · listed 2024-05-22 | 18.5% |
| [SWE-Agent / Mistral Large 2](https://swe-agent.com/latest/) | Unit test mode · coverage increase 23.0% · listed 2024-05-22 | 16.3% |
| [SWE-Agent / GPT-4](https://swe-agent.com/latest/) | Unit test mode · coverage increase 26.5% · listed 2024-05-22 | 15.9% |
| [LIBRO / GPT-4](https://arxiv.org/abs/2209.11515) | Unit test mode · coverage increase 23.8% · listed 2024-05-22 | 14.1% |
| [Aider / GPT-4](https://aider.chat) | Unit test mode · coverage increase 27.8% · listed 2024-05-22 | 12.7% |
| [SWE-Agent / Cl. 3.5 Sonnet](https://swe-agent.com/latest/) | Unit test mode · coverage increase 30.3% · listed 2024-05-22 | 12.3% |
| [SWE-Agent / GPT-4o mini](https://swe-agent.com/latest/) | Unit test mode · coverage increase 20.9% · listed 2024-05-22 | 9.8% |
| [Zero-Shot Plus / GPT-4 + BM25](https://arxiv.org/abs/2406.12952) | Unit test mode · coverage increase 21.5% · listed 2024-05-22 | 9.4% |
| [AutoCodeRover / GPT-4](https://autocoderover.dev) | Unit test mode · coverage increase 17.9% · listed 2024-05-22 | 9.1% |
| [Zero-Shot Base / GPT-4 + BM25](https://arxiv.org/abs/2406.12952) | Unit test mode · coverage increase 7.6% · listed 2024-05-22 | 3.6% |
| [SWE-Agent / Claude 3 Haiku](https://swe-agent.com/latest/) | Unit test mode · coverage increase 3.0% · listed 2024-05-22 | 2.5% |
| [SWE-Agent / Mixtral 8x22B](https://swe-agent.com/latest/) | Unit test mode · coverage increase 0.9% · listed 2024-05-22 | 0.7% |

## FAQ

### What does SWT-bench measure?

Whether an AI agent can turn a bug report into a test that proves the bug exists. For each real GitHub issue, the generated test must fail on the original code and pass after the developer's fix. SWE-bench asks for the fix; SWT-bench asks for the test.

### Which agent leads SWT-bench Verified?

Bugstack, at 91.5% with Claude Sonnet 5.5 and Claude Opus 5.5, followed by Claude Code with Claude Opus 5.5 at 87.8% and DPIAgent with GPT-5 and test selection at 87.5% (leaderboard of November 4, 2026).

### What is the difference between unit test mode and script mode on SWT-bench?

In unit test mode the agent adds tests to the project's own test suite. In reproduction script mode it writes a stand-alone script that reproduces the issue without using the test framework. The leaderboard marks script mode entries and awards its medals separately for each mode.
