# Bug Hunt Bench

> A blind-graded coding-agent benchmark with 105 planted bugs across two production TypeScript repositories.

Canonical page: https://benchlm.ai/benchmarks/bug-hunt-bench

- Category: [Coding](/coding)
- Last updated: September 23, 2026 snapshot

## About Bug Hunt Bench

- Year: 2026
- Tasks: 105 planted bugs across two production TypeScript repositories
- Format: Strict planted bugs fixed
- Difficulty: Blind production-repository bug finding and repair
- Paper: [Bug Hunt Bench method, caveats, and definitions](https://bughunt.productcompass.pm/method)

Each run gives a model the same prompt and one round per repository in its native agentic CLI. An independent judge checks the submitted diff against a withheld answer key. The score counts only planted bugs fixed; partial fixes, claimed-only findings, and genuine unplanted fixes remain separate. BenchLM mirrors the supplied 14-run effort comparison as display-only system evidence.

Bug Hunt Bench is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (14 effort-level runs)

| Rank | Model | Configuration | Creator | Score |
|------|-------|---------------|---------|-------|
| 1 | [Claude Fable 5.1](/models/claude-fable-5-1) | Claude Code · max reasoning · $87.18 list-rate estimate | Anthropic | 43 fixes |
| 2 | [GPT-5.6 Sol](/models/gpt-5-6-sol) | Codex CLI · max reasoning · $69.61 list-rate estimate | OpenAI | 42 fixes |
| 3 | [GPT-5.6 Sol](/models/gpt-5-6-sol) | Codex CLI · xhigh reasoning · $52.75 list-rate estimate | OpenAI | 39 fixes |
| 4 | [GPT-5.6 Sol](/models/gpt-5-6-sol) | Codex CLI · high reasoning · $33.92 list-rate estimate | OpenAI | 34 fixes |
| 5 | [Claude Fable 5.1](/models/claude-fable-5-1) | Claude Code · high reasoning · $48.58 list-rate estimate | Anthropic | 33 fixes |
| 6 | [Claude Fable 5.1](/models/claude-fable-5-1) | Claude Code · low reasoning · $33.00 list-rate estimate | Anthropic | 29 fixes |
| 7 | [Grok 4.6](/models/grok-4-6) | Grok Build CLI (ACP) · xhigh reasoning · $16.96 lower bound | xAI | 27 fixes |
| 8 | [Claude Opus 5](/models/claude-opus-5) | Claude Code · max reasoning · $54.94 list-rate estimate | Anthropic | 27 fixes |
| 9 | [Claude Opus 5](/models/claude-opus-5) | Claude Code · xhigh reasoning · $63.25 list-rate estimate | Anthropic | 26 fixes |
| 10 | [Claude Opus 5](/models/claude-opus-5) | Claude Code · medium reasoning · $37.40 list-rate estimate | Anthropic | 24 fixes |
| 11 | [Grok 4.6](/models/grok-4-6) | Grok Build CLI (ACP) · high reasoning · $15.73 lower bound | xAI | 23 fixes |
| 12 | [Grok 4.6](/models/grok-4-6) | Grok Build CLI (ACP) · medium reasoning · $5.80 lower bound | xAI | 22 fixes |
| 13 | [Claude Opus 5](/models/claude-opus-5) | Claude Code · high reasoning · $41.51 list-rate estimate | Anthropic | 21 fixes |
| 14 | [Grok 4.6](/models/grok-4-6) | Grok Build CLI (ACP) · low reasoning · $4.14 lower bound | xAI | 15 fixes |

## FAQ

### What does Bug Hunt Bench measure?

Bug Hunt Bench measures whether coding agents can find and fix 105 planted bugs across two production TypeScript repositories. Every run gets the same prompt and one attempt per repository. An independent judge grades the submitted diff against a withheld answer key rather than trusting the model's report.

### Which run leads the selected Bug Hunt Bench view?

Claude Fable 5.1 at max effort leads the supplied September 2, 2026 selection with 43 of 105 planted bugs fixed. GPT-5.6 Sol at max effort follows with 42. These are complete model, reasoning-effort, and CLI configurations, not isolated base-model measurements.

### Why is Bug Hunt Bench display only?

Each row combines the model with a reasoning setting, agentic CLI, one run per repository, and a cost figure that may be a list-rate estimate or reconstructed lower bound. The snapshot is useful for comparing those exact setups, but it does not enter BenchLM's model-only rankings.