A multimodal variant of SWE-bench that adds visual context (screenshots, design mockups) to software engineering issue descriptions, testing whether models can leverage visual information for code generation.
BenchLM mirrors the published score view for SWE-bench Multimodal. Claude Mythos Preview leads the public snapshot at 59%. BenchLM does not use these results to rank models overall.
Year
2025
Tasks
Multimodal software engineering tasks
Format
Code patch generation with visual context
Difficulty
Frontier multimodal coding
SWE-bench Multimodal is important because real-world software engineering increasingly involves visual inputs like UI mockups, error screenshots, and design specifications. Scores tend to be much lower than text-only SWE-bench variants.
Version
SWE-bench Multimodal 2025
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
A multimodal variant of SWE-bench that adds visual context (screenshots, design mockups) to software engineering issue descriptions, testing whether models can leverage visual information for code generation.
Claude Mythos Preview by Anthropic currently leads with a score of 59% on SWE-bench Multimodal.
1 AI models have been evaluated on SWE-bench Multimodal on BenchLM.
For engineers, researchers, and the plain curious — a weekly brief on new models, ranking shifts, and pricing changes.
Free. No spam. Unsubscribe anytime.