# ScarfBench: Cross-Framework Application Migration in Enterprise Java (ScarfBench)

> IBM Research's benchmark for coding agents that migrate complete enterprise Java applications between Jakarta EE, Quarkus and Spring while keeping them working.

Canonical page: https://benchlm.ai/benchmarks/scarfbench

- Category: [Coding](/coding)
- Last updated: Evaluated October 13, 2026 to October 17, 2026

Evaluation results published by [IBM Research](https://www.kaggle.com/benchmarks/ibm-research/scarfbench). undefined

ScarfBench asks a coding agent to migrate a complete enterprise Java application from one framework to another: Jakarta EE, Quarkus or Spring. Its 34 applications each have expert-verified versions in all three frameworks, which gives 204 directional migration tasks.

The score is Kaggle's "percent of smoke tests passed": the behavioral check that the migrated application still does what it did before, using the benchmark's expert-written tests (1,331 in the full suite). The Compile and Deploy rates in each row are the earlier gates, the share of the 204 tasks where the migrated application builds and where it starts in the target runtime.

Every row is a model inside a specific agent harness (Claude Code, Codex, Antigravity CLI, mini-SWE-agent, Pi or Kimi Code) at the reasoning setting Kaggle lists, so a gap between two rows can come from the harness as well as the model. These results are display-only and do not enter overall or category rankings.

These are published model-agent configurations. Compare the harness and reasoning setting before treating a score difference as a model difference.

- [Kaggle leaderboard and methodology](https://www.kaggle.com/benchmarks/ibm-research/scarfbench)
- [Published leaderboard data](https://www.kaggle.com/api/v1/benchmarks/ibm-research/scarfbench/leaderboard)
- [ScarfBench paper](https://arxiv.org/abs/2605.06754)
- [ScarfBench code and applications](https://github.com/scarfbench/benchmark)
- [Separate owner leaderboard (Pass@1 and Pass@3)](https://scarfbench.info/leaderboard/)
- [Apache 2.0 license shown by Kaggle](https://www.apache.org/licenses/LICENSE-2.0)

## About ScarfBench

- Year: 2026
- Tasks: 204 framework migration tasks across 34 enterprise Java applications
- Format: Percent of smoke tests passed after migration (Kaggle)
- Difficulty: Whole-application framework migration
- Paper: [ScarfBench: A Benchmark for Cross-Framework Application Migration in Enterprise Java](https://arxiv.org/abs/2605.06754)

ScarfBench (Self-Contained Application Refactoring Benchmark) has 34 applications, each with expert-verified Jakarta EE, Quarkus and Spring implementations, for 204 directional migration tasks (about 151K lines of code and 1,331 expert-written tests). Each migration must build, deploy in the target runtime, and pass behavioral tests. BenchLM mirrors the Kaggle leaderboard's percent of smoke tests passed; compile and deploy rates stay with each row on the benchmark page. Every Kaggle row is a model inside a named agent harness (Claude Code, Codex, Antigravity CLI, mini-SWE-agent, Pi or Kimi Code) at a listed reasoning setting, so BenchLM keeps the scores display-only and out of weighted rankings. IBM's separate scarfbench.info leaderboard reports Pass@1 and Pass@3 on a different protocol and is not merged.

ScarfBench is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (10 configurations)

| Rank | Model | Configuration | Creator | Score |
|------|-------|---------------|---------|-------|
| 1 | [Gemini 4 Argon](/models/gemini-4-argon) | Antigravity CLI · high reasoning · Antigravity CLI 1.3.4; Compile 98.5%; Deploy 97.5%; Evaluated 2026-10-17 | Google | 75.2% |
| 2 | [Claude Opus 5.5](/models/claude-opus-5-5) | Claude Code · max reasoning · Claude Code 2.1.280; Compile 96.6%; Deploy 95.1%; Evaluated 2026-10-17 | Anthropic | 70.5% |
| 3 | [Claude Opus 5](/models/claude-opus-5) | Claude Code · xhigh reasoning · Claude Code 2.1.273; Compile 98.5%; Deploy 97.5%; Evaluated 2026-10-13 | Anthropic | 69.8% |
| 4 | [DeepSeek V4.1 Flash 1017](/models/deepseek-v4-1-flash) | Pi · max reasoning · Pi 1.1.0; Compile 94.6%; Deploy 94.6%; Evaluated 2026-10-15 | DeepSeek | 69.0% |
| 5 | [GPT-6.1 Sol](/models/gpt-6-1-sol) | Codex · max reasoning · GPT-6.1 Sol; Compile 97.1%; Deploy 95.6%; Evaluated 2026-10-17 | OpenAI | 68.4% |
| 6 | [Gemini 3.8 Flash](/models/gemini-3-8-flash) | mini-SWE-agent · medium reasoning · mini-swe-agent 2.4.6; Compile 98.5%; Deploy 97.1%; Evaluated 2026-10-13 | Google | 66.0% |
| 7 | [GPT-6 Astra](/models/gpt-6-astra) | Codex · high reasoning · Codex 0.146.1; Compile 98.0%; Deploy 96.1%; Evaluated 2026-10-13 | OpenAI | 65.4% |
| 8 | [Kimi K3](/models/kimi-k3) | Kimi Code · max reasoning · Kimi Code 2.0.2; Compile 97.1%; Deploy 96.1%; Evaluated 2026-10-15 | Moonshot AI | 61.3% |
| 9 | [Gemini 3.1 Pro](/models/gemini-3-1-pro) | mini-SWE-agent · high reasoning · mini-swe-agent 2.4.6; Compile 90.2%; Deploy 83.3%; Evaluated 2026-10-17 | Google | 37.6% |
| 10 | [Gemma 4 26B A4B](/models/gemma-4-26b-a4b) | mini-SWE-agent · high reasoning · mini-swe-agent 2.4.6; Compile 74.0%; Deploy 68.6%; Evaluated 2026-10-17 | Google | 24.4% |

## FAQ

### What does ScarfBench measure?

IBM Research's benchmark for coding agents that migrate complete enterprise Java applications between Jakarta EE, Quarkus and Spring while keeping them working.

### Which source row has the highest reported ScarfBench value?

Gemini 4 Argon has the highest reported ScarfBench value at 75.2%. These are published model-agent configurations. Compare the harness and reasoning setting before treating a score difference as a model difference.

### How many models are evaluated on ScarfBench?

The Evaluated October 13, 2026 to October 17, 2026 contains 10 AI models.

### Does ScarfBench affect BenchLM's overall score?

Not directly. ScarfBench is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.