ScarfBench: Cross-Framework Application Migration in Enterprise Java (ScarfBench)

We show this table for reference; we do not rank on it.

IBM Research's benchmark for coding agents that migrate complete enterprise Java applications between Jakarta EE, Quarkus and Spring while keeping them working.

Evaluation results published by IBM Research. How Kaggle Benchmarks reruns tests.

Should you trust a ScarfBench score?

No caution flags. Scores are separated; the table is mirrored and not verified row by row; the benchmark is current.

Saturation Different source configurations: These are published model-agent configurations. Compare the harness and reasoning setting before treating a score difference as a model difference.
Who produced the rows Mirrored table: Every row is mirrored from Kaggle Benchmarks as it publishes it; BenchLM did not re-run these results. Source.
Freshness and weight Display only: Current, refreshed rolling. Display only; it does not move rankings. Public applications and tests with published Kaggle aggregate results.

Row statuses come from the verification registry; saturation and freshness from the benchmark lifecycle. A provider self-report is displayable evidence, not an independent run.

Smoke tests passed on ScarfBench — Evaluated October 13, 2026 to October 17, 2026

We mirror the published smoke tests passed view for ScarfBench. Gemini 4 Argon has the highest reported value at 75.2%. These are published model-agent configurations. Compare the harness and reasoning setting before treating a score difference as a model difference. We do not use these results to rank models overall.

1Closed

Gemini 4 Argon

Google

Antigravity CLI · high reasoning

Antigravity CLI 1.3.4; Compile 98.5%; Deploy 97.5%; Evaluated 2026-10-17

75.2%

Overall 82.28Context 1M

2Closed

Claude Opus 5.5

Anthropic

Claude Code · max reasoning

Claude Code 2.1.280; Compile 96.6%; Deploy 95.1%; Evaluated 2026-10-17

70.5%

Overall 86.29Context 1M

3Closed

Claude Opus 5

Anthropic

Claude Code · xhigh reasoning

Claude Code 2.1.273; Compile 98.5%; Deploy 97.5%; Evaluated 2026-10-13

69.8%

Overall 79.68Context 1M

10 configurationsCodingCurrentDisplay onlyUpdated Evaluated October 13, 2026 to October 17, 2026

Smoke tests passed table (10 configurations)

Smoke tests passed results for ScarfBench
Rank	Model / configuration	Score	Parameters (B)	Open / closed
1	Gemini 4 ArgonGoogleAntigravity CLI · high reasoningAntigravity CLI 1.3.4; Compile 98.5%; Deploy 97.5%; Evaluated 2026-10-17	75.2%	Not reported	Closed
2	Claude Opus 5.5AnthropicClaude Code · max reasoningClaude Code 2.1.280; Compile 96.6%; Deploy 95.1%; Evaluated 2026-10-17	70.5%	Not reported	Closed
3	Claude Opus 5AnthropicClaude Code · xhigh reasoningClaude Code 2.1.273; Compile 98.5%; Deploy 97.5%; Evaluated 2026-10-13	69.8%	Not reported	Closed
4	DeepSeek V4.1 Flash 1017DeepSeekPi · max reasoningPi 1.1.0; Compile 94.6%; Deploy 94.6%; Evaluated 2026-10-15	69.0%	Not reported	Open
5	GPT-6.1 SolOpenAICodex · max reasoningGPT-6.1 Sol; Compile 97.1%; Deploy 95.6%; Evaluated 2026-10-17	68.4%	Not reported	Closed
6	Gemini 3.8 FlashGooglemini-SWE-agent · medium reasoningmini-swe-agent 2.4.6; Compile 98.5%; Deploy 97.1%; Evaluated 2026-10-13	66.0%	Not reported	Closed
7	GPT-6 AstraOpenAICodex · high reasoningCodex 0.146.1; Compile 98.0%; Deploy 96.1%; Evaluated 2026-10-13	65.4%	Not reported	Closed
8	Kimi K3Moonshot AIKimi Code · max reasoningKimi Code 2.0.2; Compile 97.1%; Deploy 96.1%; Evaluated 2026-10-15	61.3%	Not reported	Open
9	Gemini 3.1 ProGooglemini-SWE-agent · high reasoningmini-swe-agent 2.4.6; Compile 90.2%; Deploy 83.3%; Evaluated 2026-10-17	37.6%	Not reported	Closed
10	Gemma 4 26B A4BGooglemini-SWE-agent · high reasoningmini-swe-agent 2.4.6; Compile 74.0%; Deploy 68.6%; Evaluated 2026-10-17	24.4%	Not reported	Open

How to read these results

ScarfBench asks a coding agent to migrate a complete enterprise Java application from one framework to another: Jakarta EE, Quarkus or Spring. Its 34 applications each have expert-verified versions in all three frameworks, which gives 204 directional migration tasks.

The score is Kaggle's "percent of smoke tests passed": the behavioral check that the migrated application still does what it did before, using the benchmark's expert-written tests (1,331 in the full suite). The Compile and Deploy rates in each row are the earlier gates, the share of the 204 tasks where the migrated application builds and where it starts in the target runtime.

Every row is a model inside a specific agent harness (Claude Code, Codex, Antigravity CLI, mini-SWE-agent, Pi or Kimi Code) at the reasoning setting Kaggle lists, so a gap between two rows can come from the harness as well as the model. These results are display-only and do not enter overall or category rankings.

Snapshot

10 configurations204 migration tasks1,331 expert-written testsDisplay only

Kaggle leaderboard and methodology Published leaderboard data ScarfBench paper ScarfBench code and applications Separate owner leaderboard (Pass@1 and Pass@3)Apache 2.0 license shown by Kaggle

These are published model-agent configurations. Compare the harness and reasoning setting before treating a score difference as a model difference. The table retains each source configuration and its reported value.

10 configurations are shown for ScarfBench. The benchmark falls in the Coding category. ScarfBench is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About ScarfBench

Year

2026

Tasks

204 framework migration tasks across 34 enterprise Java applications

Format

Percent of smoke tests passed after migration (Kaggle)

Difficulty

Whole-application framework migration

ScarfBench (Self-Contained Application Refactoring Benchmark) has 34 applications, each with expert-verified Jakarta EE, Quarkus and Spring implementations, for 204 directional migration tasks (about 151K lines of code and 1,331 expert-written tests). Each migration must build, deploy in the target runtime, and pass behavioral tests. BenchLM mirrors the Kaggle leaderboard's percent of smoke tests passed; compile and deploy rates stay with each row on the benchmark page. Every Kaggle row is a model inside a named agent harness (Claude Code, Codex, Antigravity CLI, mini-SWE-agent, Pi or Kimi Code) at a listed reasoning setting, so BenchLM keeps the scores display-only and out of weighted rankings. IBM's separate scarfbench.info leaderboard reports Pass@1 and Pass@3 on a different protocol and is not merged.

ScarfBench: A Benchmark for Cross-Framework Application Migration in Enterprise Java Public benchmark source

Freshness and provenance

Version

ScarfBench v1 (Kaggle)

Refresh cadence

Rolling

Staleness state

Current

Question availability

Public applications and tests with published Kaggle aggregate results

CurrentDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does ScarfBench measure?

IBM Research's benchmark for coding agents that migrate complete enterprise Java applications between Jakarta EE, Quarkus and Spring while keeping them working.

Which source row has the highest reported ScarfBench value?

Gemini 4 Argon has the highest reported ScarfBench value at 75.2%. These are published model-agent configurations. Compare the harness and reasoning setting before treating a score difference as a model difference. BenchLM shows this benchmark for display only and does not use it in overall rankings.

How many models are evaluated on ScarfBench?

The Evaluated October 13, 2026 to October 17, 2026 snapshot contains 10 AI models.

Last updated: Evaluated October 13, 2026 to October 17, 2026 · mirrored from the public benchmark leaderboard

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 6,100+ subscribers.

One email each week. Unsubscribe anytime.