Vals SWE-bench Multimodal evaluation (SWE-bench Multimodal (Vals))
We show this table for reference; we do not rank on it.
JavaScript issue resolution where bug reports depend on screenshots, evaluated by Vals AI with no agent internet access.
Evaluation results published by Vals AI. Published leaderboard aggregates and evaluation settings only. Dataset contents are not included.
Resolved issues on SWE-bench Multimodal (Vals) — October 13, 2026
We mirror the published resolved issues view for SWE-bench Multimodal (Vals). Claude Opus 5.5 leads the public snapshot at 70.21%, followed by Claude Fable 5.1 (61.25%) and Claude Sonnet 5.5 (58.33%). We do not use these results to rank models overall.
Claude Opus 5.5
Anthropic
Claude Fable 5.1
Anthropic
Claude Sonnet 5.5
Anthropic
18 modelsCodingCurrentDisplay onlyUpdated October 13, 2026
Resolved issues table (18 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Claude Opus 5.5Anthropic | 70.21% | Not reported | Closed |
| 2 | Claude Fable 5.1Anthropic | 61.25% | Not reported | Closed |
| 3 | Claude Sonnet 5.5Anthropic | 58.33% | Not reported | Closed |
| 4 | Gemini 4 ArgonGooglehigh reasoning | 56.04% | Not reported | Closed |
| 5 | Muse Wonder FullMetamax reasoning | 47.29% | Not reported | Closed |
| 6 | DeepSeek V4.1 Flash 1017DeepSeekmax reasoning | 46.46% | Not reported | Open |
| 7 | Muse WonderMetamax reasoning | 45.42% | Not reported | Closed |
| 8 | Claude Haiku 5.5Anthropic | 43.96% | Not reported | Closed |
| 9 | GPT-6.1 SolOpenAImax reasoning | 43.54% | Not reported | Closed |
| 10 | GPT-6 AstraOpenAImax reasoning | 43.33% | Not reported | Closed |
| 11 | Muse Spark 1.3 MaxMetamax reasoning | 43.13% | Not reported | Unknown |
| 12 | Grok 4.7xAIxhigh reasoning | 42.92% | Not reported | Closed |
| 13 | MiMo-V2.6-ProXiaomi | 41.25% | Not reported | Open |
| 14 | Gemini 3.8 FlashGooglehigh reasoning | 40.83% | Not reported | Closed |
| 15 | Step 5 PreviewStepFun | 40.83% | Not reported | Open |
| 16 | Qwen3.8 MaxAlibaba | 39.17% | Not reported | Open |
| 17 | GPT-6 LunaOpenAImax reasoning | 37.50% | Not reported | Closed |
| 18 | Kimi K3Moonshot AImax reasoning | 35.83% | Not reported | Pending |
How SWE-bench Multimodal (Vals) is shown here
We mirror Vals AI's SWE-bench Multimodal results for JavaScript issues whose reports depend on screenshots. Each model runs once with mini-swe-agent across 480 tasks. Screenshots are downloaded before the run, and the agent has no internet access.
These source-specific runs remain separate from text-only SWE-bench and provider launch results. They are display only and do not enter model rankings.
Snapshot
The published SWE-bench Multimodal (Vals) snapshot places Claude Opus 5.5 first at 70.21%. The third row is 11.88 points behind. The broader top-10 range is 26.88 points, so the table still separates the published systems.
18 models have been evaluated on SWE-bench Multimodal (Vals). The benchmark falls in the Coding category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. SWE-bench Multimodal (Vals) is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About SWE-bench Multimodal (Vals)
Year
2026
Tasks
480 JavaScript issues with visual bug reports
Format
Resolved issue percentage
Difficulty
Visual software issue resolution
Vals AI evaluates 480 tasks across eleven repositories with mini-swe-agent, one trial per model, screenshots supplied before the run, and no agent internet access. These source-specific results stay display only and remain separate from text-only SWE-bench scores.
Freshness and provenance
Version
SWE-bench Multimodal (Vals) 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does SWE-bench Multimodal (Vals) measure?
JavaScript issue resolution where bug reports depend on screenshots, evaluated by Vals AI with no agent internet access.
Which model leads the published SWE-bench Multimodal (Vals) snapshot?
Claude Opus 5.5 currently leads the published SWE-bench Multimodal (Vals) snapshot with 70.21% resolved issues. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on SWE-bench Multimodal (Vals)?
The October 13, 2026 snapshot contains 18 AI models.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 5,600+ readers.
One email each week. Unsubscribe anytime.