Bug Hunt Bench

We show this table for reference; we do not rank on it.

A blind-graded coding-agent benchmark with 105 planted bugs across two production TypeScript repositories.

Planted bugs fixed on Bug Hunt Bench — November 13, 2026 snapshot

We mirror the published planted bugs fixed view for Bug Hunt Bench. Claude Fable 5.2 leads the public snapshot at 58.0 fixes, followed by Gemini 4 Carbon (57.3 fixes) and Gemini 4 Carbon (56.0 fixes). We do not use these results to rank models overall.

1Closed

Claude Fable 5.2

Anthropic

Claude Code · max reasoning

mean of 3 runs · $139.65 list-rate estimate

58.0 fixes

Overall 93.65Context 1M

2Closed

Gemini 4 Carbon

Google

Antigravity CLI · high reasoning

mean of 3 runs · $31.32 list-rate estimate

57.3 fixes

Overall 89.15Context 1M

3Closed

Gemini 4 Carbon

Google

Antigravity CLI · medium reasoning

mean of 3 runs · $24.95 list-rate estimate

56.0 fixes

Overall 89.15Context 1M

28 featured configurationsCodingCurrentDisplay onlyUpdated November 13, 2026 snapshot

Planted bugs fixed table (28 featured configurations)

Planted bugs fixed results for Bug Hunt Bench
Rank	Model / configuration	Score	Parameters (B)	Open / closed
1	Claude Fable 5.2AnthropicClaude Code · max reasoningmean of 3 runs · $139.65 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	58.0 fixes	Not reported	Closed
2	Gemini 4 CarbonGoogleAntigravity CLI · high reasoningmean of 3 runs · $31.32 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	57.3 fixes	Not reported	Closed
3	Gemini 4 CarbonGoogleAntigravity CLI · medium reasoningmean of 3 runs · $24.95 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	56.0 fixes	Not reported	Closed
4	Claude Sonnet 5.5AnthropicClaude Code · max reasoningmean of 3 runs · $154.03 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	51.3 fixes	Not reported	Closed
5	Claude Fable 5.2AnthropicClaude Code · xhigh reasoningmean of 3 runs · $61.01 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	49.0 fixes	Not reported	Closed
6	GPT-6 AstraOpenAICodex CLI · max reasoningmean of 3 runs · $33.03 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	45.0 fixes	Not reported	Closed
7	GPT-6.1 SolOpenAICodex CLI · max reasoningmean of 3 runs · $5.88 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	44.3 fixes	Not reported	Closed
8	GPT-6 AstraOpenAICodex CLI · xhigh reasoningsingle run · $24.22 list-rate estimate	43.0 fixes	Not reported	Closed
9	Claude Fable 5.1AnthropicClaude Code · max reasoningsingle run · $87.18 list-rate estimate	43.0 fixes	Not reported	Closed
10	DeepSeek V4.1 Flash 1017DeepSeekDeepSeek Harness / DeepSeek API · max reasoningmean of 7 runs · $1.84 billed Run caveats Mean of 7 runs. Read the linked source for observed spread and limits specific to this configuration.	42.7 fixes	Not reported	Open
11	GPT-6.1 SolOpenAICodex CLI · xhigh reasoningmean of 3 runs · $4.35 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	42.7 fixes	Not reported	Closed
12	Muse Wonder FullMetaMuse Code / Meta API · max reasoningmean of 2 runs Run caveats Mean of 2 runs. Read the linked source for observed spread and limits specific to this configuration.	42.0 fixes	Not reported	Closed
13	Claude Opus 5.5AnthropicClaude Code · max reasoningmean of 3 runs · $58.53 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	41.7 fixes	Not reported	Closed
14	Muse Wonder FullMetaMuse Code / Meta API · xhigh reasoningmean of 2 runs Run caveats Mean of 2 runs. Read the linked source for observed spread and limits specific to this configuration.	41.5 fixes	Not reported	Closed
15	Gemini 4 ArgonGoogleAntigravity CLI · high reasoningmean of 3 runs · $7.16 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	40.7 fixes	Not reported	Closed
16	DeepSeek V4.1 Flash 1017DeepSeekDeepSeek Harness / DeepSeek API · high reasoningmean of 5 runs · $1.27 billed Run caveats Mean of 5 runs. Read the linked source for observed spread and limits specific to this configuration.	39.2 fixes	Not reported	Open
17	DeepSeek V4.1 Flash 1017DeepSeekClaude Code / DeepSeek API · max reasoningmean of 7 runs · $1.01 billed Run caveats Mean of 7 runs. Read the linked source for observed spread and limits specific to this configuration.	39.1 fixes	Not reported	Open
18	Kimi K3.1 ProMoonshot AIClaude Code / Moonshot API · max reasoningmean of 3 runs · $13.76 billed Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	36.0 fixes	Not reported	Open
19	Claude Opus 5.5AnthropicClaude Code · xhigh reasoningmean of 3 runs · $34.98 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	36.0 fixes	Not reported	Closed
20	Claude Sonnet 5.5AnthropicClaude Code · xhigh reasoningmean of 3 runs · $42.86 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	36.0 fixes	Not reported	Closed
21	Gemini 4 HydrogenGoogleAntigravity CLI · xhigh reasoningmean of 3 runs · $7.07 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	32.7 fixes	Not reported	Closed
22	Muse Spark 1.3MetaMuse Code / Meta API · max reasoningmean of 5 runs · $18.11 list-rate estimate Run caveats Mean of 5 runs. Read the linked source for observed spread and limits specific to this configuration.	32.2 fixes	Not reported	Closed
23	Muse WonderMetaMuse Code / Meta API · max reasoningmean of 3 runs · $27.87 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	31.7 fixes	Not reported	Closed
24	Grok 4.7xAIGrok Build CLI (ACP) · xhigh reasoningmean of 5 runs · $22.96 lower bound Run caveats Mean of 5 runs. Read the linked source for observed spread and limits specific to this configuration.	29.0 fixes	Not reported	Closed
25	Gemini 4 HydrogenGoogleAntigravity CLI · high reasoningmean of 3 runs · $4.42 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	28.3 fixes	Not reported	Closed
26	Qwen3.8 MaxAlibabaClaude Code / Alibaba API · max reasoningmean of 3 runs · $26.76 list-rate estimate Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	25.7 fixes	Not reported	Open
27	MiMo-V2.6-ProXiaomiClaude Code / OpenRouter · default reasoningmean of 3 runs · $0.86 billed Run caveats Mean of 3 runs. Read the linked source for observed spread and limits specific to this configuration.	22.7 fixes	Not reported	Open
28	Claude Haiku 5.5AnthropicClaude Code · max reasoningmean of 2 runs · $5.83 list-rate estimate Run caveats Mean of 2 runs. Read the linked source for observed spread and limits specific to this configuration.	21.5 fixes	Not reported	Closed

How we show Bug Hunt Bench

The publisher's featured view compares 28 configurations across 18 models. Each run receives one round on two production TypeScript repositories containing 105 planted bugs, and an independent judge grades the submitted diff against a withheld answer key.

Only verified fixes to planted bugs enter the score. Partial fixes, claimed-only findings, and genuine unplanted fixes stay separate. Across all 332 source runs, 23 bugs remain unfixed by every model.

Repeated configurations report the published mean, with the run count beside each row. Single runs stay labeled, and per-row caveats preserve observed variation. Costs retain their billed, list-rate, or lower-bound basis.

Snapshot

28 featured configurations18 models105 planted bugs2 production repositories23 bugs still unfixedDisplay only

Featured Bug Hunt Bench view Public benchmark JSON Method, caveats, and definitions Raw scoreboards and prompts

The published Bug Hunt Bench snapshot places Claude Fable 5.2 first at 58.0 fixes. The third row is 2.0 points behind. The broader top-10 range is 15.3 points, so the table still separates the published systems.

28 featured configurations are shown for Bug Hunt Bench. The benchmark falls in the Coding category. Bug Hunt Bench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About Bug Hunt Bench

Year

2026

Tasks

105 planted bugs across two production TypeScript repositories

Format

Strict planted bugs fixed

Difficulty

Blind production-repository bug finding and repair

Each run gives a model the same prompt and one round per repository in its native agentic CLI. An independent judge checks the submitted diff against a withheld answer key. The score counts only planted bugs fixed; partial fixes, claimed-only findings, and genuine unplanted fixes remain separate. We mirror the publisher's featured configurations, retaining repeat-run means and single runs as display-only system evidence.

Bug Hunt Bench method, caveats, and definitions Public benchmark source

Freshness and provenance

Version

Bug Hunt Bench 2026

Refresh cadence

Quarterly

Staleness state

Current

Question availability

Public benchmark set

CurrentDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does Bug Hunt Bench measure?

Bug Hunt Bench measures whether coding agents can find and fix 105 planted bugs across two production TypeScript repositories. Every run gets the same prompt and one attempt per repository. An independent judge grades the submitted diff against a withheld answer key rather than trusting the model's report.

How are repeated Bug Hunt Bench runs reported?

A configuration tested more than once reports the publisher's mean planted-bug count, including fractional values. The table labels each mean with its run count and keeps single runs separate. Per-row caveats preserve the observed spread; a small difference between configurations does not establish a reliable lead.

Why is Bug Hunt Bench display only?

Each row combines a model, reasoning setting, agentic CLI, and one attempt per repository per run. Repeated configurations report means; costs may be billed amounts, list-rate estimates, or lower bounds. These exact setups help compare debugging workflows, but their scores do not enter our overall or category rankings.

Last updated: November 13, 2026 snapshot · mirrored from the public benchmark leaderboard

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 7,000+ subscribers.

One email each week. Unsubscribe anytime.