# AI Benchmarks Reference Guide > Comprehensive reference for all 639 benchmarks tracked on BenchLM AI. ## Agentic Benchmarks ### ExplorationBench AlienCode (ExplorationBench AlienCode Best@3) Tests whether a system can discover a programming language's hidden rules through experiments and apply them to unseen tasks. - **Year**: 2026 - **Tasks**: 70 held-out program-synthesis tasks; 31 discovery targets - **Format**: Held-out accuracy (Best@3) - **Difficulty**: Discovering and applying unfamiliar executable rules ### ExplorationBench AlienLogic (ExplorationBench AlienLogic Best@3) Tests whether a system can discover altered logic rules and use them to prove or correctly decline unseen theorems. - **Year**: 2026 - **Tasks**: 70 held-out theorems; 24 altered rules - **Format**: Held-out accuracy (Best@3) - **Difficulty**: Learning and applying unfamiliar inference rules ### Terminal-Bench-Science 0.1 (6x verifier timeout) (Terminal-Bench-Science 0.1 with sixfold verifier timeout) Google-run scientific workflow evaluation using a sixfold verifier timeout to address verification timeouts. - **Year**: 2026 - **Tasks**: 70 expert-curated scientific research workflows - **Format**: Resolution rate across 3 trials per task - **Difficulty**: Frontier scientific research workflows ### CWE-bench v1 (CWE-bench v1) Defensive vulnerability patching on 120 held-out tasks spanning 73 weakness types. - **Year**: 2026 - **Tasks**: Audit and patch held-out C/C++, Go, Java, TypeScript/JavaScript, Python, and Rust repositories - **Format**: Pass@1 - **Difficulty**: Frontier agent evaluation ### DRACO (Data Research and Analysis with Complex Operations) Agentic data-analysis tasks scored against per-task rubrics at a 980K-token budget. - **Year**: 2026 - **Tasks**: Agentic data research and analysis tasks - **Format**: Normalized rubric score - **Difficulty**: Professional data analysis ### MCP-Atlas claim coverage (MCP-Atlas mean claim coverage) Average coverage of required claims in answers produced during real-world MCP tool-use workflows. - **Year**: 2026 - **Tasks**: Production-like multi-server MCP workflows - **Format**: Mean claim coverage - **Difficulty**: Real-world tool use ### LAB all-pass (Anthropic harness) (Legal Agent Benchmark all-pass rate — Anthropic harness) Strict task success requiring every expert-written legal-work rubric criterion to pass. - **Year**: 2026 - **Tasks**: 1,235 legal-agent tasks - **Format**: All-criteria pass rate - **Difficulty**: Professional legal work ### LAB criterion-pass (Anthropic harness) (Legal Agent Benchmark mean criterion-pass rate — Anthropic harness) Mean fraction of expert-written rubric criteria passed across legal-agent tasks. - **Year**: 2026 - **Tasks**: 1,235 legal-agent tasks - **Format**: Mean criterion-pass rate - **Difficulty**: Professional legal work ### LAB all-pass (Harvey held-out) (Legal Agent Benchmark all-pass rate — Harvey held-out set) Harvey AI's strict held-out task success rate requiring every rubric criterion to pass. - **Year**: 2026 - **Tasks**: Harvey-held-out legal-agent tasks - **Format**: All-criteria pass rate - **Difficulty**: Professional legal work ### LAB criterion-pass (Harvey held-out) (Legal Agent Benchmark mean criterion-pass rate — Harvey held-out set) Harvey AI's mean criterion-level score on its held-out legal-agent evaluation. - **Year**: 2026 - **Tasks**: Harvey-held-out legal-agent tasks - **Format**: Mean criterion-pass rate - **Difficulty**: Professional legal work ### Toolathlon Verified Pass@3 (Toolathlon Verified Pass@3) Fraction of Toolathlon Verified tasks solved in at least one of three trials. - **Year**: 2026 - **Tasks**: 108 verified real-world tool-use tasks - **Format**: Pass@3 - **Difficulty**: Long-horizon application tool use ### Toolathlon Verified Pass³ (Toolathlon Verified Pass cubed) Fraction of Toolathlon Verified tasks solved in all three independent trials. - **Year**: 2026 - **Tasks**: 108 verified real-world tool-use tasks - **Format**: All-three-trials pass rate - **Difficulty**: Long-horizon application tool use ### Toolathlon Verified avg. turns (Toolathlon Verified average assistant turns) Average assistant turns per Toolathlon Verified trajectory. - **Year**: 2026 - **Tasks**: 108 verified real-world tool-use tasks - **Format**: Average trajectory length - **Difficulty**: Long-horizon application tool use ### Terminal-Bench 3.0 (Terminal-Bench 3.0) A continuously maintained benchmark for difficult computer work, including coding, deep learning, finance, engineering, math, and science tasks. - **Year**: 2026 - **Tasks**: 74 professional computer-work tasks across 7 domains - **Format**: Task completion rate - **Difficulty**: Frontier autonomous knowledge work ### Terminal-Bench 4.0 (Terminal-Bench 4.0) The current Terminal-Bench release measures difficult computer work after recalibrating task resources, fixing unstable tasks, and removing tasks that no longer separate frontier systems. - **Year**: 2026 - **Tasks**: 66 professional computer-work tasks - **Format**: Task completion rate across 5 trials per task - **Difficulty**: Frontier autonomous knowledge work ### Terminal-Bench-Science 0.1 (Terminal-Bench-Science 0.1) A benchmark of AI agents completing expert-curated research workflows across the life, physical, Earth, mathematical, and engineering sciences. - **Year**: 2026 - **Tasks**: 70 expert-curated scientific research workflows - **Format**: Resolution rate across 3 trials per task - **Difficulty**: Frontier scientific research workflows ### EQ-Bench 4 (EQ-Bench 4) A multi-turn benchmark of emotional and social intelligence using synthetic personas and pairwise LLM judging. - **Year**: 2026 - **Tasks**: Up to 120 multi-turn persona scenarios per model - **Format**: Pairwise Elo - **Difficulty**: Emotional and social intelligence ### AutoCAD-Bench (AutoCAD-Bench) A Markov Studios computer-use benchmark that asks agents to produce 2D drawings and 3D models in AutoCAD. - **Year**: 2026 - **Tasks**: 21 2D drawing tasks and 29 3D modeling tasks - **Format**: Task completion rate at a 75-point rubric threshold - **Difficulty**: Professional CAD computer use ### Design Arena Agentic Web Dev (Design Arena Agentic Web Dev Elo) A display-only Elo rating from blinded comparisons of multi-file web applications built by coding agents. - **Year**: 2026 - **Tasks**: Multi-file web application development - **Format**: Elo from blinded human preferences - **Difficulty**: Agentic frontend development ### AA Briefcase (Artificial Analysis Briefcase) An independently evaluated professional-work benchmark reported as Elo. - **Year**: 2026 - **Tasks**: Professional knowledge-work tasks - **Format**: Elo - **Difficulty**: Professional work ### AA AutomationBench (Artificial Analysis AutomationBench) An independently evaluated automation benchmark from Artificial Analysis. - **Year**: 2026 - **Tasks**: Business-process automation tasks - **Format**: Task success rate - **Difficulty**: Agentic automation ### AA EnterpriseOps-Gym (Artificial Analysis EnterpriseOps-Gym) An independently evaluated enterprise-operations benchmark from Artificial Analysis. - **Year**: 2026 - **Tasks**: Enterprise operations workflows - **Format**: Task success rate - **Difficulty**: Enterprise agent operations ### AA Harvey LAB v1.0 (Artificial Analysis Harvey LAB-AA v1.0) Archived v1.0 legal-agent results from Artificial Analysis, kept separate from v1.1. - **Year**: 2026 - **Tasks**: Legal agent tasks - **Format**: Published v1.0 score - **Difficulty**: Professional legal work ### Harvey LAB-AA v1.1 (Harvey LAB-AA v1.1) Legal-agent task success requiring every rubric criterion to pass and no material hallucination, averaged across three judges. - **Year**: 2026 - **Tasks**: 120 private legal tasks across 24 practice areas - **Format**: Hallucination-Gated All-Pass Rate - **Difficulty**: Professional legal work ### Harvey LAB-AA v1.1 Criterion Pass (Harvey LAB-AA v1.1 Criterion Pass) The share of legal-deliverable rubric criteria passed, pooled across three judges before the hallucination gate. - **Year**: 2026 - **Tasks**: 120 private legal tasks across 24 practice areas - **Format**: Criterion Pass Rate - **Difficulty**: Professional legal work ### AA ITBench (Artificial Analysis ITBench-AA) An independently evaluated IT-operations benchmark from Artificial Analysis. - **Year**: 2026 - **Tasks**: IT incident-response tasks - **Format**: Task success rate - **Difficulty**: Enterprise IT operations ### AA Tau3 Banking (Artificial Analysis Tau3-Banking) An independently evaluated Tau3 banking benchmark from Artificial Analysis. - **Year**: 2026 - **Tasks**: Banking tool-use workflows - **Format**: Task success rate - **Difficulty**: Agentic banking workflows ### Terminal-Bench 2.0 (Terminal-Bench 2.0) A benchmark for agentic software engineering tasks executed in real terminal environments. Models must inspect files, run commands, edit code, and recover from errors over multi-step workflows. - **Year**: 2026 - **Tasks**: Terminal-based software tasks - **Format**: Interactive CLI agent evaluation - **Difficulty**: Professional software engineering ### Terminal-Bench 2.1 (Terminal-Bench 2.1) Terminal-Bench 2.1 results, mostly provider-reported, stored separately from the Terminal-Bench 2.0 lane. - **Year**: 2026 - **Tasks**: Terminal-based software-agent tasks - **Format**: Interactive task success rate - **Difficulty**: Professional software engineering ### BrowseComp (BrowseComp) BrowseComp tests whether a browsing agent can find difficult, fact-seeking answers across the web. Its 1,266 problems use short answers that are easier to grade than an open-ended research report. - **Year**: 2025 - **Tasks**: Research questions requiring browsing - **Format**: Web search and evidence synthesis - **Difficulty**: Hard web research ### HLE w/ tools (Humanity's Last Exam with tools) Tool-augmented Humanity's Last Exam scores reported in DeepSeek-V4 thinking-mode evaluations. - **Year**: 2026 - **Tasks**: Expert questions with tool use - **Format**: Pass@1 - **Difficulty**: Frontier tool-augmented reasoning ### GDPval-AA (GDPval-AA) An agentic real-world work-task evaluation reported as an Elo score in DeepSeek-V4 thinking-mode evaluations. - **Year**: 2026 - **Tasks**: Agentic real-world work tasks - **Format**: Elo - **Difficulty**: Professional agentic workflows ### GDPval-AA (GDPval-AA normalized) An Artificial Analysis normalized score for economically valuable tasks. - **Year**: 2026 - **Tasks**: Economically valuable tasks - **Format**: Normalized score - **Difficulty**: Professional agentic workflows ### AA Agentic Index (Artificial Analysis Agentic Index) A display-only Artificial Analysis agentic index. - **Year**: 2026 - **Tasks**: Cross-benchmark agentic index - **Format**: Aggregated model score - **Difficulty**: Display-only external reference ### GDP.pdf (Artificial Analysis GDP.pdf) Artificial Analysis' document-output benchmark: professional tasks whose deliverable is a produced PDF, one of the ten components of its Intelligence Index v4.3. - **Year**: 2026 - **Tasks**: Professional document-production tasks - **Format**: Task success rate - **Difficulty**: Professional knowledge work ### AA-AnalystAgent (Artificial Analysis AnalystAgent) Artificial Analysis' data-analysis benchmark, testing agents on spreadsheet and document work to answer the quantitative questions a business or data analyst faces day to day. - **Year**: 2026 - **Tasks**: Spreadsheet and document analysis questions - **Format**: Task success rate - **Difficulty**: Business and data analysis ### APEX-Agents-AA (APEX-Agents-AA) Artificial Analysis' implementation of the APEX-Agents benchmark for long-horizon professional-services agent tasks. - **Year**: 2026 - **Tasks**: 452 professional-services agent tasks - **Format**: Pass@1 - **Difficulty**: Long-horizon workplace agent tasks ### Gert Labs (Gert Labs Composite Game Benchmark) A game-environment benchmark that evaluates AI models in novel games covering strategic planning, resource management, spatial reasoning, cooperation, and theory of mind. - **Year**: 2026 - **Tasks**: Novel game environments - **Format**: Composite game leaderboard - **Difficulty**: Agentic coding and decision-making ### OSWorld-Verified (OSWorld-Verified) OSWorld-Verified is the July 2025 repaired release of OSWorld's real-computer evaluation. It measures whether a model-agent system can finish desktop and web tasks from configured starting states, with success checked by execution-based evaluators. - **Year**: 2025 - **Tasks**: 369 real-world computer tasks (361 when eight Google Drive tasks are excluded) - **Format**: Execution-based interactive task success - **Difficulty**: Multi-step desktop and cross-application workflows ### OSWorld 2.0 (OSWorld 2.0) OSWorld 2.0 evaluates long-horizon computer-use tasks. Published results use different task releases, metrics, and run settings, so scores appear by comparable protocol and unmatched rows stay unranked. - **Year**: 2026 - **Tasks**: 108 long-horizon computer-use workflows - **Format**: Interactive computer-use evaluation; binary completion and partial-score metrics - **Difficulty**: Long-horizon professional workflows ### CyberGym (CyberGym) A cybersecurity task benchmark for evaluating defensive cyber workflows and vulnerability-oriented agent performance. - **Year**: 2026 - **Tasks**: 1,507 vulnerability analysis instances - **Format**: Vulnerability reproduction and PoC generation - **Difficulty**: Real-world cybersecurity ### CWE-Bench (CWE-Bench) An external benchmark for evaluating whether coding agents can produce correct patches for real-world software vulnerabilities. - **Year**: 2026 - **Tasks**: Real-world vulnerability patching tasks - **Format**: Pass@1 - **Difficulty**: Automated vulnerability remediation ### CTI-REALM (CTI-REALM) A cybersecurity benchmark that measures whether an agent can turn raw threat-intelligence reports into working detection rules. - **Year**: 2026 - **Tasks**: Threat-intelligence-to-detection-rule workflows - **Format**: Success rate - **Difficulty**: Professional cyber threat detection ### Cybench (Cybench) A cybersecurity benchmark of professional Capture the Flag tasks for measuring autonomous cyber agent capability and risk. - **Year**: 2025 - **Tasks**: 40 professional CTF tasks - **Format**: Cybersecurity agent task completion - **Difficulty**: Professional cybersecurity ### ExploitGym (ExploitGym) A controlled benchmark for evaluating whether AI agents can extend vulnerability-triggering inputs into working exploits. - **Year**: 2026 - **Tasks**: 898 exploitation tasks - **Format**: Working exploit generation - **Difficulty**: Advanced cybersecurity exploitation ### JobBench (JobBench) An occupational agent benchmark for professional workflows that workers say they most want delegated to AI. - **Year**: 2026 - **Tasks**: 130 tasks across 35 occupations - **Format**: Agentic workplace deliverables - **Difficulty**: Professional multi-source workflows ### BrowseComp-VL (BrowseComp-VL) A vision-language browsing benchmark for multimodal web research and tool-use workflows. - **Year**: 2026 - **Tasks**: Multimodal browsing tasks - **Format**: Vision-language web research evaluation - **Difficulty**: Multimodal browser-agent ### OSWorld (OSWorld) A computer-use benchmark for GUI task completion across the broader OSWorld task suite. - **Year**: 2026 - **Tasks**: Computer-use tasks - **Format**: Interactive GUI evaluation - **Difficulty**: Broad computer-use suite ### AndroidWorld (AndroidWorld) A mobile GUI agent benchmark for completing Android app workflows and on-device tasks. - **Year**: 2026 - **Tasks**: Android app workflows - **Format**: Interactive mobile-agent evaluation - **Difficulty**: Complex mobile task completion ### WebVoyager (WebVoyager) A browser-agent benchmark for completing multi-step workflows on live websites. - **Year**: 2026 - **Tasks**: Live website workflows - **Format**: Interactive browser-agent evaluation - **Difficulty**: Multi-step web navigation ### MCP Atlas (MCP Atlas) A benchmark for tool-calling over Model Context Protocol integrations and external tools. - **Year**: 2026 - **Tasks**: Tool-integrated agent tasks - **Format**: Interactive tool-calling evaluation - **Difficulty**: Advanced tool use ### Kimi Claw 24/7 (Kimi Claw 24/7 Bench) A Moonshot AI internal long-horizon agent benchmark for persistent professional coworking tasks. - **Year**: 2026 - **Tasks**: 17 professional scenarios, 610 evaluation points - **Format**: Average pass rate across repeated OpenClaw runs - **Difficulty**: Long-horizon agentic work ### MCP Mark Verified (MCPMark-Verified) A human-verified edition of MCPMark for MCP tool use across Notion, GitHub, Filesystem, Postgres, and Playwright server environments. - **Year**: 2026 - **Tasks**: MCP tool-use tasks across five server environments - **Format**: Interactive MCP task completion - **Difficulty**: Advanced tool use ### Toolathlon (Toolathlon) A tool-use benchmark focused on selecting, sequencing, and completing tasks with external tools. - **Year**: 2026 - **Tasks**: Multi-tool workflows - **Format**: Interactive tool-calling evaluation - **Difficulty**: Advanced tool use ### Toolathlon-Verified (Toolathlon-Verified) A verified tool-use benchmark variant for completing multi-step workflows with external tools. - **Year**: 2026 - **Tasks**: Verified multi-tool workflows - **Format**: Interactive tool-use score - **Difficulty**: Advanced tool use ### AutomationBench (AutomationBench) An agent benchmark for completing automation workflows in reproducible task environments. - **Year**: 2026 - **Tasks**: 600 public automation tasks - **Format**: Agent task-completion score - **Difficulty**: Long-horizon automation ### AutomationBench (Zapier 1.0.6) (AutomationBench, Zapier private evaluation set release 1.0.6) An agent benchmark for completing business workflows across simulated applications and API endpoints. - **Year**: 2026 - **Tasks**: 600+ held-out workflows across 47 simulated applications and six business domains - **Format**: Task pass rate - **Difficulty**: Long-horizon automation ### Agents' Last Exam (Agents' Last Exam) An agent benchmark reported in DeepSeek's V4 Flash 0731 launch comparison. - **Year**: 2026 - **Tasks**: Agent tasks - **Format**: Provider-reported task score - **Difficulty**: Advanced agentic work ### APEX-Agents (APEX-Agents) A professional-services agent benchmark covering long-horizon knowledge-work tasks. - **Year**: 2026 - **Tasks**: Professional-services agent tasks - **Format**: Agent task-completion score - **Difficulty**: Long-horizon professional work ### SpreadsheetBench 2 (SpreadsheetBench 2) A spreadsheet-focused benchmark for agentic analysis and editing workflows. - **Year**: 2026 - **Tasks**: Spreadsheet analysis and editing tasks - **Format**: Agent task-completion score - **Difficulty**: Professional spreadsheet work ### DECK-Bench (DECK-Bench (Internal)) Moonshot AI's internal benchmark for presentation and deck-production workflows. - **Year**: 2026 - **Tasks**: Internal presentation workflows - **Format**: Internal evaluation score - **Difficulty**: Professional presentation creation ### ZClawBench (ZClawBench) A Z.AI benchmark for OpenClaw-style agent workflows spanning information search, office work, data analysis, development and operations, automation, and security. - **Year**: 2026 - **Tasks**: OpenClaw agent workflows - **Format**: End-to-end agent benchmark - **Difficulty**: Broad productivity and operations workflows ### τ²-bench results (τ²-Bench Tool-Agent-User Evaluation) This route is a sourced ledger for published τ²-bench results. Most current rows come from Artificial Analysis's telecom implementation, while named provider rows can use telecom, airline, retail, or aggregate setups. - **Year**: 2025 - **Tasks**: Airline, retail, and telecom customer-service task sets - **Format**: Published domain success or pass^k results - **Difficulty**: Dual-control customer-service workflows ### DeepSearchQA (DeepSearchQA) An agentic browsing benchmark where models search the web, gather evidence, and answer list-style questions using browser tools. - **Year**: 2026 - **Tasks**: Agentic browsing and list-answer questions - **Format**: Search / open / find browser-agent evaluation - **Difficulty**: Agentic web research ### τ²-bench Airline (τ²-Bench Airline Domain) τ²-bench Airline tests conversational agents on airline customer-service tasks governed by domain policy and database-changing tools. - **Year**: 2025 - **Tasks**: Airline customer-service tasks - **Format**: Domain success under a published trial policy - **Difficulty**: Policy-constrained airline support workflows ### PinchBench (PinchBench) An OpenClaw agent benchmark from Kilo that measures successful task completion across standardized real-world agent workflows. - **Year**: 2026 - **Tasks**: 23 OpenClaw agent tasks - **Format**: Average success rate from official runs - **Difficulty**: Long-horizon agent workflows ### OpenHands Index (OpenHands Index) A holistic coding-agent benchmark that evaluates AI agents across issue resolution, frontend work, greenfield development, testing, and information gathering. - **Year**: 2025 - **Tasks**: SWE-bench Verified, SWE-bench Multimodal, Commit0, SWT-bench Verified, and GAIA - **Format**: Macro-average across five coding-agent categories - **Difficulty**: Real-world software engineering agent tasks ### SWE-Atlas Refactoring (SWE-Atlas Refactoring) A Scale SWE-Atlas software-engineering agent benchmark focused on refactoring tasks. - **Year**: 2026 - **Tasks**: SWE-Atlas refactoring tasks - **Format**: Refactoring score with confidence intervals - **Difficulty**: Real-world software-engineering agent tasks ### SWE-sweep (SWE-sweep: Can Agents Autonomously Find and Fix Bugs?) Tests whether coding agents can find and fix multiple bugs across a repository without issue descriptions, while preserving existing behavior. - **Year**: 2026 - **Tasks**: 4,068 reference bugs across 100 repositories in 22 languages - **Format**: Micro-average percentage of bugs resolved with a repository regression gate - **Difficulty**: Repository-wide bug discovery and repair ### SWE Refactor Bench (SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?) Tests whether coding agents can complete long-horizon, whole-repository stack migrations while preserving the original program's behavior. - **Year**: 2026 - **Tasks**: 20 whole-repository stack migrations - **Format**: Migration audit, frozen behavioral checks, and agentic verification - **Difficulty**: 6- to 30-hour autonomous repository migrations ### AI4AI-Bench (AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement) Tests whether coding agents can improve the training algorithm inside an existing AI research codebase, then survive a sealed training run and held-out evaluation. - **Year**: 2026 - **Tasks**: 10 AI training-algorithm design tasks - **Format**: Four-hour code rewrite followed by sealed training and held-out evaluation - **Difficulty**: End-to-end AI research and algorithm design ### InferenceBench (InferenceBench) A benchmark for open-ended LLM inference optimization by AI agents. Agents receive a base model, one H100, and a fixed time budget to build a valid OpenAI-compatible inference server that improves serving speed. - **Year**: 2026 - **Tasks**: 4 inference-serving optimization scenarios - **Format**: Two-hour autonomous CLI agent run - **Difficulty**: Open-ended ML systems engineering ### EdgeBench (EdgeBench) A ByteDance Seed benchmark of 134 real-world, day-scale tasks that measures how autonomous agents learn from environment feedback over 12+ hour interaction horizons, spanning scientific and ML, systems and software engineering, optimization, knowledge, formal, and game domains. - **Year**: 2026 - **Tasks**: 134 tasks (51 public) across 6 domains - **Format**: Long-horizon interactive agent evaluation - **Difficulty**: Day-scale expert tasks ### BFCL v4 (Berkeley Function Calling Leaderboard v4) A function-calling benchmark for tool selection, schema adherence, and argument correctness. - **Year**: 2025 - **Tasks**: Function-calling tasks - **Format**: Tool invocation and schema evaluation - **Difficulty**: Advanced tool use ### MLE-Bench Lite (MLE-Bench Lite) A lightweight machine-learning competition benchmark that measures whether models can iteratively train, evaluate, and improve ML systems in low-resource settings. - **Year**: 2026 - **Tasks**: Low-resource ML competitions - **Format**: Autonomous iterative ML optimization - **Difficulty**: Agentic machine learning ### MM-ClawBench (MM-ClawBench) An OpenClaw-derived agent benchmark covering practical work and life tasks such as office document delivery, research, planning, and code maintenance. - **Year**: 2026 - **Tasks**: OpenClaw-style real-world tasks - **Format**: Agent workflow evaluation - **Difficulty**: Broad real-world agentic execution ### Claw-Eval (Claw-Eval) A transparent real-world autonomous-agent benchmark with 300 human-verified tasks, 2,159 rubric items, and Pass^3 scoring across general, multi-turn, and native multimodal agent tasks. - **Year**: 2026 - **Tasks**: 300 tasks, 2,159 rubrics - **Format**: End-to-end autonomous-agent evaluation with Pass^3 scoring - **Difficulty**: Real-world general, multi-turn, and native multimodal agent execution ### ResearchClawBench (ResearchClawBench) An end-to-end autonomous scientific research benchmark with 40 tasks across 10 scientific domains, where agents receive related literature and raw data, then attempt to rediscover the hidden target paper. - **Year**: 2026 - **Tasks**: 40 tasks across 10 scientific domains - **Format**: End-to-end autonomous research evaluation with RADS scoring - **Difficulty**: Scientific research re-discovery ### QwenClawBench (QwenClawBench) Qwen's internal OpenClaw-style benchmark for measuring broad real-world agent performance across practical productivity and research tasks. - **Year**: 2026 - **Tasks**: Real-world agent workflows - **Format**: End-to-end agent evaluation - **Difficulty**: Broad real-world agentic execution ### QwenWebBench (QwenWebBench) A Qwen benchmark for artifact and webpage generation quality reported as an Elo-style rating. - **Year**: 2026 - **Tasks**: Web artifacts and interactive deliverables - **Format**: Elo-style artifact benchmark - **Difficulty**: Artifact generation ### τ³-bench results (τ³-Bench Tool-Agent-User Evaluation) τ³-bench is the current evolution of Sierra's tool-agent-user framework, adding corrected task releases and newer knowledge and voice evaluation modes alongside airline, retail, and telecom. - **Year**: 2026 - **Tasks**: Corrected customer-service tasks plus knowledge and voice evaluation modes - **Format**: Published domain or average success results - **Difficulty**: Long-horizon, multimodal, and knowledge-aware tool use ### VITA-Bench (VITA-Bench) An interactive real-world agent benchmark grounded in practical consumer-service tasks such as delivery, in-store consumption, and online travel workflows. - **Year**: 2025 - **Tasks**: Interactive consumer-service agent tasks - **Format**: End-to-end interactive agent evaluation - **Difficulty**: Long-horizon real-world workflows ### DeepPlanning (DeepPlanning) A long-horizon planning benchmark that tests whether agents can optimize under explicit time, budget, and feasibility constraints. - **Year**: 2026 - **Tasks**: Travel planning and constrained shopping - **Format**: Long-horizon planning benchmark - **Difficulty**: Constrained agent planning ### MCP-Tasks (MCP-Tasks) A Model Context Protocol task benchmark used in Qwen's launch tables to measure practical execution over MCP-style tools and integrations. - **Year**: 2026 - **Tasks**: MCP-integrated tool tasks - **Format**: Interactive tool-use evaluation - **Difficulty**: Advanced MCP workflows ### WideResearch (WideResearch) A broad research-agent benchmark for open-ended information gathering, synthesis, and answer construction across wide search spaces. - **Year**: 2026 - **Tasks**: Open-ended research tasks - **Format**: Multi-source research evaluation - **Difficulty**: Broad research-agent workflows ### CoWorkBench (CoWorkBench) Qwen's internal benchmark for long-horizon professional work across computer science, finance, law, medicine, and other productivity domains. - **Year**: 2026 - **Tasks**: Long-horizon professional workflows - **Format**: Provider-run agent score - **Difficulty**: Cross-domain professional work ### MobileWorld (MobileWorld) A mobile-use agent benchmark for completing interactive tasks in smartphone environments. - **Year**: 2026 - **Tasks**: Interactive mobile-device workflows - **Format**: Mobile agent task score - **Difficulty**: Long-horizon mobile computer use ### GAIA (General AI Assistants) GAIA evaluates AI models on real-world tasks that are conceptually simple for humans but require multi-step reasoning, web browsing, tool use, and multimodal understanding for AI. Tasks span three difficulty levels and test practical assistant capabilities rather than academic knowledge. - **Year**: 2024 - **Tasks**: 466 ### TAU-bench (Tool-Agent-User Benchmark) Original TAU-bench tests an agent's tool use and policy compliance in simulated airline and retail conversations. Its pass^k metric asks whether the agent succeeds in every one of k repeated trials. - **Year**: 2024 - **Tasks**: Airline and retail task sets in the archived 2024 release - **Format**: Domain-specific pass^1 through pass^4 task success - **Difficulty**: Policy-constrained, multi-turn customer service ### WebArena (WebArena Web Agent Benchmark) WebArena tests whether a browser-agent system can complete 812 long-horizon tasks inside self-hosted replicas of functional websites. It checks the requested end state, so a result reflects the model, agent scaffold, browser interface, action budget, and evaluator together—not the base model alone. - **Year**: 2024 - **Tasks**: 812 long-horizon browser tasks - **Format**: End-state task success - **Difficulty**: Stateful multi-site browser work ### WebArena-Verified (WebArena-Verified Browser Agent Benchmark) WebArena-Verified is an audited release of the WebArena browser-agent benchmark. It rechecks task descriptions, reference answers, and evaluators, and replaces nondeterministic judging with deterministic checks where possible. - **Year**: 2025 - **Tasks**: 812 verified tasks; separate 258-task Hard subset - **Format**: Deterministic end-state task success - **Difficulty**: Audited stateful browser work ### MEWC (Multi-Environment Web Challenge) A benchmark that evaluates AI agents on multi-environment web challenges, testing navigation and task completion across diverse live web environments. - **Year**: 2026 - **Tasks**: Web-agent tasks - **Format**: Browser task completion - **Difficulty**: Open-web agent workflows ### Finance Agent v2 (Finance Agent v2) Vals AI benchmark for realistic financial analyst agent tasks across qualitative analysis, quantitative analysis, market work, comparables, precedents, earnings, disclosure, and modeling. - **Year**: 2026 - **Tasks**: Financial analyst task categories - **Format**: Mean score across repeated runs - **Difficulty**: Professional expert-task agent workflow ### Market-Bench (Market-Bench) A quantitative-trading implementation benchmark that asks models to build backtesters under market-book liquidity and execution-delay constraints, then compares their outputs with a verifier. - **Year**: 2025 - **Tasks**: 3 quantitative-trading strategies - **Format**: Backtester implementation scored by mean absolute error - **Difficulty**: Market simulation and quantitative coding ### GDPval rubrics (GDPval rubrics) A display-only provider-table GDPval rubric score for economically valuable work tasks. - **Year**: 2026 - **Tasks**: Economically valuable work tasks - **Format**: Rubric score - **Difficulty**: Professional agentic workflows ### BankerToolBench (BankerToolBench) A display-only provider benchmark for finance-oriented tool-use and agent workflows. - **Year**: 2026 - **Tasks**: Finance and banking tool-use tasks - **Format**: Task success rate - **Difficulty**: Professional finance-agent workflows ### Hermes Index (Hermes Index: agentic capability and cost per task inside Hermes Agent) An index of agentic capability from Nous Research. It averages pass@1 scores on four suites, Hermes Bench, TerminalBench 4, TerminalBench Science and SkillsBench, with every run inside the Hermes Agent harness, and reports the mean cost per task beside the score. - **Year**: 2026 - **Tasks**: Four suites: Hermes Bench (150 tasks), TerminalBench 4 without its 4 GPU tasks, TerminalBench Science and SkillsBench - **Format**: Mean pass@1 score across four suites, with mean cost per task - **Difficulty**: Frontier agentic ### Hermes Bench (Hermes Bench: 150 agent tasks graded on the files and state the agent leaves behind) Tests agents on 150 tasks across 25 categories, including Hermes skills, research, diagrams and art, memory, tool use and safety. Each agent starts in a workspace with real files, some tasks add follow-up turns, and the grader checks the files and state left behind. - **Year**: 2026 - **Tasks**: 150 tasks across 25 categories - **Format**: pass@1 score - **Difficulty**: Frontier agentic ### ApprenticeBench (ApprenticeBench: end-to-end computer use, continual learning, and long-horizon agency on a real accounts-payable job) Tests whether a computer-use agent can learn a real accounts-payable job on the job, processing 100 vendor bills in a company ERP system with only the handbook, historical records, and mentor feedback a new hire would get. - **Year**: 2026 - **Tasks**: 100 vendor bills processed in sequence inside a simulated construction company - **Format**: Cumulative success rate over 100 bills - **Difficulty**: Long-horizon computer use with offline and online continual learning ### MazeBench (MazeBench: visual spatial reasoning in a 3D open world) Tests whether a coding agent can explore a 256-room 3D puzzle world and collect its 100 hidden gems, using box-pushing puzzles, camera rotation, and long-horizon planning. - **Year**: 2026 - **Tasks**: 100 hidden gems across 256 rooms in one open world - **Format**: Gems collected out of 100, with rooms visited as a second score - **Difficulty**: Long-horizon spatial planning, with single puzzles that can take over one hundred moves ### BFCL v3 (Berkeley Function Calling Leaderboard v3) A function-calling benchmark for tool selection, schema adherence, and argument correctness, covering single-turn, parallel, irrelevance and multi-turn subsets. - **Year**: 2025 - **Tasks**: Function-calling tasks - **Format**: Tool invocation and schema evaluation - **Difficulty**: Advanced tool use ### ReviewBench (ReviewBench) Measures how reliably code-review agents identify useful issues in real pull requests while avoiding false positives. - **Year**: 2026 - **Tasks**: 219 pull requests across 187 repositories and 19 languages - **Format**: Grounded precision, recall, and F1; augmented metrics ### Ramp Accounting Bench (Ramp Accounting Bench) A Ramp Labs public leaderboard for long-horizon accounting agent work. BenchLM shows it as display-only reference data and excludes it from model rankings. - **Year**: 2026 - **Tasks**: 137 accounting tasks · 3 attempts per task - **Format**: Mean criterion-level score across three attempts - **Difficulty**: Long-horizon accounting agent evaluation ### HiL-Bench (Scale Labs HiL-Bench) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 27 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### MCP Atlas (Scale Labs MCP Atlas) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 34 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### Remote Labor Index (Scale Labs Remote Labor Index) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 20 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### Visual Tool Bench (Scale Labs Visual Tool Bench) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 21 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### Robo-bench (Vals AI Robo-bench) Tests agents controlling a simulated robot arm and humanoid through motor commands, sensor readings, and camera images. - **Year**: 2026 - **Tasks**: 24 simulated robot-control tasks - **Format**: Mean task score out of 100 - **Difficulty**: Real-time perception, planning, and motor control ### SciUniverse Level 1 (SciUniverse Level 1) SciUniverse Level 1 tests whether models carry out scientific work across chemistry, biology, and materials science. Its 92 tasks include physical experiments, simulated lab work, and measurement analysis, with Pass@1 averaged equally across 17 task families. - **Year**: 2026 - **Tasks**: 92 tasks in 17 task families - **Format**: Scientific work through a facility harness; Pass@1 - **Difficulty**: Level 1 scientific tasks ### Parallel Search Capability (Parallel Search Capability Leaderboard) A 29-model comparison of how well models use Parallel Search Fast and Extract to answer web research questions. - **Year**: 2026 - **Tasks**: 100 questions each from DeepSearchQA, HLE, and WISER - **Format**: Equal-weight Search Intelligence Score (0–100) - **Difficulty**: Multi-step web research and specialist questions ### Vals RSI Index (Vals RSI Index v1.1) Tests autonomous AI research on compression, language-model training, judge-harness engineering, and post-training under fixed compute and time budgets. - **Year**: 2026 - **Tasks**: 4 autonomous research tasks - **Format**: Mean reference-anchored log score, expressed as a percentage - **Difficulty**: AI research under fixed compute and time budgets ### Web Search Index (Vals Web Search Index) A Vals AI comparison of native provider search and Exa across finance-analysis and legal-research tasks. - **Year**: 2026 - **Tasks**: Finance Agent Benchmark v2 and Legal Research Benchmark tasks - **Format**: Accuracy by model and search-tool combination - **Difficulty**: Professional web research with controlled search-tool variants ### Time Horizon Index: KSP (Vals Time Horizon Index: Kerbal Space Program) A Vals AI agent benchmark that gives each system five days to build and run a space program in Kerbal Space Program. - **Year**: 2026 - **Tasks**: 30 progressively harder Kerbal Space Program missions - **Format**: Mission-ladder progress with partial credit - **Difficulty**: Long-horizon autonomous computer use and planning ### CUA-bench (Vals CUA-bench) A Vals AI computer-use benchmark in which agents play six commercial video games with a keyboard and mouse. - **Year**: 2026 - **Tasks**: Six commercial video-game control tasks, including held-out games - **Format**: Overall score with game-level task splits - **Difficulty**: Keyboard-and-mouse computer use in real-time games ### Finance Agent v1.1 (Vals Finance Agent v1.1) An archived Vals benchmark covering retrieval, numerical reasoning, financial modeling, market analysis, earnings, trends, and adjustments. - **Year**: 2026 - **Tasks**: 11 financial analyst task views - **Format**: Accuracy with task-level breakdowns - **Difficulty**: Professional financial analysis ### SRE Bench (Vals SRE Bench (formerly ReverseEngBench)) A contamination-resistant agent benchmark for reverse engineering real-world binaries. - **Year**: 2026 - **Tasks**: Real-world binary reverse-engineering tasks - **Format**: Fully solved rate and capability score - **Difficulty**: Agentic reverse engineering ### Vals Terminal-Bench 1.0 mirror (Vals-hosted Terminal-Bench 1.0 mirror) A Vals-hosted view of Terminal-Bench 1.0 with easy, medium, and hard task splits. - **Year**: 2026 - **Tasks**: Terminal tasks split by easy, medium, and hard difficulty - **Format**: Accuracy score - **Difficulty**: Terminal-agent execution ### SRE-Bench (SRE-Bench binary reverse engineering) A contamination-controlled benchmark that asks agents to reverse engineer compiled binaries without source access and satisfy six task-specific objectives per challenge. - **Year**: 2026 - **Tasks**: 262 binary reverse-engineering instances - **Format**: Fully solved challenge rate (pass@1) - **Difficulty**: Binary reverse engineering ### ExploitBench (ExploitBench v8-bench) A cybersecurity benchmark for evaluating LLM agents on full-control V8 exploit synthesis using 16 measured exploit capability flags. - **Year**: 2026 - **Tasks**: V8 exploit synthesis runs - **Format**: Capability coverage percentage over 16 flags - **Difficulty**: Browser exploitation and cybersecurity ### ACE solved (ACE Cyber Range Challenges Solved) Number of advanced cyber-range challenges solved in the joint NIST CAISI and UK AISI preliminary evaluation. - **Year**: 2026 - **Tasks**: 41 advanced cyber-range challenges - **Format**: Challenges solved - **Difficulty**: Advanced cyber operations ### The Last Ones steps (The Last Ones Average Progress) Average step reached on a 32-step long-horizon cyber range. - **Year**: 2026 - **Tasks**: 32-step long-horizon cyber range - **Format**: Average step reached - **Difficulty**: Long-horizon cyber operations ### The Last Ones completion (The Last Ones Cyber Range Completion Rate) Share of runs that completed the 32-step cyber range within the 100-million-token limit. - **Year**: 2026 - **Tasks**: 10 long-horizon runs - **Format**: Completion rate - **Difficulty**: Long-horizon cyber operations ### SEC-Bench Pro (SEC-Bench Pro) Cybersecurity benchmark for agentic vulnerability analysis and exploit-oriented security tasks. - **Year**: 2026 - **Tasks**: Security engineering tasks - **Format**: Success rate - **Difficulty**: Advanced cybersecurity ### FrontierCyber (FrontierCyber) Independent evaluation of AI agents against vulnerable real-world systems in dynamic environments. - **Year**: 2026 - **Tasks**: 197 dynamic cyber tasks - **Format**: Tasks solved - **Difficulty**: Easy through elite cyber operations ### CyScenarioBench success (CyScenarioBench Average Success Rate) Average success rate across realistic, long-horizon cybersecurity scenarios. - **Year**: 2026 - **Tasks**: 11 cyber scenarios - **Format**: Average success rate - **Difficulty**: Long-horizon cybersecurity ### CyScenarioBench solved (CyScenarioBench Scenarios Ever Solved) Number of CyScenarioBench scenarios completed successfully in at least one run. - **Year**: 2026 - **Tasks**: 11 cyber scenarios - **Format**: Scenarios solved at least once - **Difficulty**: Long-horizon cybersecurity ### Atomic network attacks (Atomic Network Attack Simulation) Irregular's domain-level evaluation of network attack simulation capability. - **Year**: 2026 - **Tasks**: Atomic cyber tasks - **Format**: Domain average - **Difficulty**: Network attack simulation ### Atomic vulnerability research (Atomic Vulnerability Research and Exploitation) Irregular's domain-level evaluation of vulnerability research and exploitation capability. - **Year**: 2026 - **Tasks**: Atomic cyber tasks - **Format**: Domain average - **Difficulty**: Vulnerability research and exploitation ### Atomic evasion (Atomic Evasion) Irregular's domain-level evaluation of cybersecurity evasion capability. - **Year**: 2026 - **Tasks**: Atomic cyber tasks - **Format**: Domain average - **Difficulty**: Cybersecurity evasion ### SCONE post-cutoff success (SCONE Post-Cutoff Exploit Success) Share of SCONE smart-contract vulnerabilities exploited on the 12-task post-cutoff set. - **Year**: 2026 - **Tasks**: 12 post-cutoff smart-contract vulnerabilities - **Format**: Best@8 exploit success rate - **Difficulty**: Smart-contract exploitation ### SCONE simulated revenue (SCONE Post-Cutoff Simulated Exploit Revenue) Simulated value captured across the SCONE post-cutoff smart-contract set. - **Year**: 2026 - **Tasks**: 12 post-cutoff smart-contract vulnerabilities - **Format**: Simulated USD millions, Best@8 - **Difficulty**: Smart-contract exploitation ### Firefox 147 exploits (Firefox 147 Working Exploit Rate) Share of patched Firefox 147 JavaScript-engine targets for which the model produced a working arbitrary-code-execution exploit. - **Year**: 2026 - **Tasks**: 250 Firefox 147 vulnerability trials - **Format**: Working arbitrary-code-execution rate - **Difficulty**: Browser exploit development ### Anthropic OSS-Fuzz crash (Anthropic OSS-Fuzz Any-Crash Rate) Share of evaluated OSS-Fuzz entry points where the model produced at least a crash. - **Year**: 2026 - **Tasks**: Approximately 830 OSS-Fuzz entry points - **Format**: Any-crash rate - **Difficulty**: Vulnerability discovery ### Anthropic OSS-Fuzz write primitive (Anthropic OSS-Fuzz Write-Primitive-or-Higher Rate) Share of evaluated OSS-Fuzz entry points where the model achieved a write primitive or stronger result. - **Year**: 2026 - **Tasks**: Approximately 830 OSS-Fuzz entry points - **Format**: Write-primitive-or-higher rate - **Difficulty**: Vulnerability exploitation ### CVE-Bench zero-day (CVE-Bench v1 Zero-Day Black-Box Evaluation) OpenAI's black-box, no-source variant of CVE-Bench v1 across 40 critical vulnerabilities. - **Year**: 2026 - **Tasks**: 40 critical CVEs - **Format**: Pass@1 over three rollouts - **Difficulty**: Black-box vulnerability exploitation ### ALE-Bench (Agents Last Exam) A benchmark for agentic professional workflows with verifiable success criteria, reporting pass rates and partial scores for model plus agent-harness rows. - **Year**: 2026 - **Tasks**: 152 ALE-V1 professional workflow tasks across 13 top-level domains - **Format**: Pass rate, partial-credit score, cost, token, and duration metadata - **Difficulty**: Real-world agentic workflows ### RuneScape-Bench (RuneBench / runescape-bench) An agentic coding benchmark where models use a TypeScript SDK to play a RuneScape-like environment and optimize skill-training performance. - **Year**: 2026 - **Tasks**: 16 RuneScape skill-training tasks - **Format**: Average log XP-rate score - **Difficulty**: Agentic gameplay automation ### Toloka Arena (Toloka Arena) An independent agentic-intelligence evaluation from Toloka using private simulated workflows and a pass^5 metric. - **Year**: 2026 - **Tasks**: Private simulated enterprise workflows - **Format**: pass^5 arena score - **Difficulty**: Agentic workflow reliability ### EMB (Vals EMB) Evaluating agents on Excel-based financial modeling tasks - **Year**: 2026 - **Tasks**: Excel-based financial modeling tasks - **Format**: Accuracy score - **Difficulty**: Professional finance modeling ### CyberBench (Vals CyberBench) Can autonomous agents craft PoC inputs that trigger OSS-Fuzz vulnerabilities—and stop crashing after the fix? - **Year**: 2026 - **Tasks**: OSS-Fuzz PoC and patch-verification tasks - **Format**: Accuracy score - **Difficulty**: Autonomous cybersecurity exploit reproduction ### BioMysteryBench (Vals) (Vals BioMysteryBench agent leaderboard) Independent agent runs on biological data investigations, split by human-solvable and human-difficult tasks. - **Year**: 2026 - **Tasks**: End-to-end bioinformatics investigations from raw data - **Format**: Average task accuracy across three runs - **Difficulty**: Agentic biological analysis ### Tax Agent Bench (Vals Tax Agent Bench) Agent evaluation on research-grade US tax questions using the Tax Agent Bench harness. - **Year**: 2026 - **Tasks**: US tax research, calculations, forms, filings, and precedent analysis - **Format**: Overall score; all-pass score shown separately - **Difficulty**: Research-grade US tax questions ### Harvey's Legal Agent Benchmark (Vals Harvey's Legal Agent Benchmark) Tests an agent's ability to complete legal work using documents, spreadsheets, presentations, and file-system tools - **Year**: 2026 - **Tasks**: Legal agent work across documents, spreadsheets, presentations, and files - **Format**: Accuracy score - **Difficulty**: Professional legal workflow automation ### Legal Research Bench (Vals Legal Research Bench) Evaluating agents on legal research tasks across diverse areas of US law - **Year**: 2026 - **Tasks**: US-law legal research tasks - **Format**: Accuracy score - **Difficulty**: Professional legal research ### SkillsBench (Vals SkillsBench) How important are skills for agents? - **Year**: 2026 - **Tasks**: Agent skill-importance tasks - **Format**: Accuracy score - **Difficulty**: Agent skill evaluation ### Agent Poker Bench (Vals Agent Poker Bench) Which model can make the most money playing poker? - **Year**: 2026 - **Tasks**: Poker-playing agent trials - **Format**: Accuracy score - **Difficulty**: Strategic game-agent decision making ### Public Benefits Bench v1.1 (Vals Public Benefits Bench v1.1) Can AI help people navigate SNAP benefits? - **Year**: 2026 - **Tasks**: SNAP public-benefits navigation tasks - **Format**: Accuracy score - **Difficulty**: Public-benefits policy navigation ### Public Benefits Bench v1 (Vals Public Benefits Bench v1) Can AI help people navigate SNAP benefits? - **Year**: 2026 - **Tasks**: SNAP public-benefits navigation tasks - **Format**: Accuracy score - **Difficulty**: Public-benefits policy navigation ### DiG-bench (DiG-bench: Discovery in Games) Tests scientific discovery by asking agents to infer undisclosed rules across 70 interactive text-based games. - **Year**: 2026 - **Tasks**: 70 interactive games across seven difficulty tiers - **Format**: Expected composite score with confidence interval - **Difficulty**: Unknown-rule discovery and interactive reasoning ### Vending-Bench 2 (Vending-Bench 2) Tests long-horizon coherence by asking agents to operate a simulated vending-machine business for one year. - **Year**: 2026 - **Tasks**: One year of simulated autonomous business operation per run - **Format**: Arithmetic mean final bank balance in USD - **Difficulty**: Long-horizon agent coherence and business operation ### MysteryMechanism (Vals MysteryMechanism) Can agents rediscover sealed mathematical mechanisms through bounded experiments? - **Year**: 2026 - **Tasks**: Sealed mathematical mechanisms probed under an experiment budget - **Format**: Accuracy score - **Difficulty**: Agentic scientific discovery ### APEX-Agents 1.1 (Mercor) (APEX-Agents 1.1 — Mercor evaluation) Professional-services tasks across investment banking, management consulting, and corporate law. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 240 tasks in the version 1.1 revision - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### APEX-Accounting (Mercor) (APEX-Accounting — Mercor evaluation) Month-end accounting tasks that require work across files and applications. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 160 held-out accounting tasks - **Format**: Mean Score - **Difficulty**: Source-specific evaluation ### BrowseComp Extended (Mercor) (BrowseComp Extended — Mercor evaluation) Finding difficult information through web research. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 100 Mercor tasks; 130 tasks in the separately reported original set - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### GDPval Extended (Mercor) (GDPval Extended — Mercor evaluation) Producing workplace deliverables from reference files. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 132 Mercor tasks; 213 tasks in the separately reported original set - **Format**: Mean Score - **Difficulty**: Source-specific evaluation ### Terminal-Bench 2.1 Extended (Mercor) (Terminal-Bench 2.1 Extended — Mercor evaluation) Completing tasks in command-line environments. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 99 Mercor tasks; 89 tasks in the separately reported original set - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### GDPval (Mercor) (GDPval — Mercor evaluation) Producing workplace deliverables from reference files. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 213 source-reported tasks - **Format**: Mean Score - **Difficulty**: Source-specific evaluation ### DeepResearch Bench II (Mercor) (DeepResearch Bench II — Mercor evaluation) Writing research reports from web evidence. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 130 source-reported tasks - **Format**: Mean Score - **Difficulty**: Source-specific evaluation ### Harvey LAB (Mercor) (Harvey LAB — Mercor evaluation) Producing legal work across practice areas and contract tasks. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 1,749 source-reported tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### RedlineBench (Mercor) (RedlineBench — Mercor evaluation) Negotiating and revising contracts across multiple turns. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 140 source-reported tasks - **Format**: Turn-Weighted Mean Score - **Difficulty**: Source-specific evaluation ### Tau3-Banking (Mercor) (Tau3-Banking — Mercor evaluation) Handling simulated banking-support conversations with tools. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 97 source-reported tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ## Coding Benchmarks ### PostTrainBench v1.1 (PostTrainBench v1.1) Post-training four base language models across seven weighted benchmarks, with ten hours and one H100 per run. - **Year**: 2026 - **Tasks**: Post-training Qwen3 1.7B and 4B, SmolLM3 3B, and Gemma3 4B - **Format**: Weighted aggregate across four base models and seven benchmarks - **Difficulty**: Frontier agent evaluation ### ProgramBench (episode 1) (ProgramBench hidden-test pass rate after episode 1) Program-reconstruction hidden-test pass rate after the first of five sequential long-context episodes. - **Year**: 2026 - **Tasks**: 166 golden program-reconstruction tasks - **Format**: Hidden-test pass rate after episode 1 - **Difficulty**: Long-context clean-room software engineering ### AA LiveCodeBench (Artificial Analysis LiveCodeBench) An independently evaluated LiveCodeBench result from Artificial Analysis. - **Year**: 2026 - **Tasks**: Contamination-resistant coding tasks - **Format**: Pass rate - **Difficulty**: Competitive programming ### Terminal-Bench 2.1 (Terminal-Bench 2.1) Terminal-Bench 2.1 results, mostly provider-reported, stored separately from the Terminal-Bench 2.0 lane. - **Year**: 2026 - **Tasks**: Terminal-based software-agent tasks - **Format**: Interactive task success rate - **Difficulty**: Professional software engineering ### HumanEval (Evaluating Large Language Models Trained on Code) HumanEval tests whether generated Python functions pass unit tests on 164 handwritten problems. These scores describe function-level correctness; they do not measure repository work or a coding agent. - **Year**: 2021 - **Tasks**: 164 problems - **Format**: Python function generation - **Difficulty**: Introductory to intermediate programming ### BigCodeBench (BigCodeBench) A code-generation benchmark reported in DeepSeek-V4 base-model evaluations. - **Year**: 2026 - **Tasks**: Code generation tasks - **Format**: Pass@1 - **Difficulty**: Software engineering ### Codeforces (Codeforces Rating) Competitive-programming rating reported for DeepSeek-V4 thinking-mode evaluations. - **Year**: 2026 - **Tasks**: Competitive programming contests - **Format**: Rating - **Difficulty**: Elite competitive programming ### Terminal-Bench 2.0 (Terminal-Bench 2.0) A benchmark for agentic software engineering tasks executed in real terminal environments. DeepSeek reports it in the agentic section, while BenchLM also mirrors it in coding for models that publish it as a developer-task signal. - **Year**: 2026 - **Tasks**: Terminal-based software tasks - **Format**: Interactive CLI agent evaluation - **Difficulty**: Professional software engineering ### SWE-bench Verified (Software Engineering Benchmark Verified) SWE-bench Verified tests patches for 500 human-filtered GitHub issues. A score measures how many tasks an evaluated model-and-agent setup resolved; the model name alone does not identify that setup. - **Year**: 2024 - **Tasks**: 500 verified issues - **Format**: Code patch generation - **Difficulty**: Professional software engineering ### SWE-Rebench (SWE-Rebench) A continuously updated software engineering benchmark by Nebius using fresh GitHub issues to avoid contamination. Models are evaluated 5 times per problem under a fixed ReAct scaffolding; the Resolved Rate (best pass@1) is reported. - **Year**: 2026 - **Tasks**: Fresh GitHub issues (rolling window) - **Format**: Code patch generation - **Difficulty**: Professional software engineering ### LiveCodeBench (LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code) A continuously updated coding benchmark built from newly collected LeetCode, AtCoder, and Codeforces problems. Fresh problem windows reduce one contamination path, but results still need a release and setup check. - **Year**: 2024 - **Tasks**: Continuously updated contest problems - **Format**: Competitive-programming evaluation - **Difficulty**: Competitive programming level ### LiveCodeBench v6 (LiveCodeBench v6) LiveCodeBench v6 is a named release slice used in provider comparison tables. Keeping it separate prevents v6 results from being mixed into older or rolling LiveCodeBench windows. - **Year**: 2026 - **Tasks**: Fresh programming problems - **Format**: Provider-published v6 competitive programming results - **Difficulty**: Competitive programming level ### LiveCodeBench v5 (LiveCodeBench v5) LiveCodeBench v5 is a named release and date-window slice. BenchLM keeps explicitly labeled v5 rows outside the rolling weighted lane. - **Year**: 2025 - **Tasks**: July 2024 to May 2025 release window - **Format**: Provider-published v5 competitive programming result - **Difficulty**: Competitive programming level ### LiveCodeBench Pass@1-COT (LiveCodeBench Pass@1 with Chain-of-Thought) This lane contains DeepSeek's LiveCodeBench Pass@1-COT results. The explicit metric and prompting label keeps them separate from generic and version-specific LiveCodeBench rows. - **Year**: 2026 - **Tasks**: DeepSeek-V4 report evaluation window - **Format**: Pass@1-COT competitive programming results - **Difficulty**: Competitive programming level ### LiveCodeBench Pro (LiveCodeBench Pro) A harder competitive-programming benchmark family built from Codeforces, ICPC, and IOI problems, with quarter-specific public leaderboards and difficulty-aware reporting. - **Year**: 2025 - **Tasks**: Quarter-specific contest programming sets - **Format**: Competitive programming - **Difficulty**: High-end contest programming ### FLTEval (FLTEval) A repository-level Lean 4 proof engineering benchmark that measures whether a model can complete formal proofs and correctly define new mathematical concepts inside realistic FLT project pull requests. - **Year**: 2026 - **Tasks**: FLT project pull requests - **Format**: Lean 4 repository task completion - **Difficulty**: Formal verification / proof engineering ### SWE-bench Pro (SWE-bench Pro) A long-horizon repository benchmark built to test realistic software engineering work. Its scores need a task-quality and setup check before they support a coding-agent decision. - **Year**: 2025 - **Tasks**: 1,865 repository problems - **Format**: Repository task completion - **Difficulty**: Long-horizon professional engineering ### Senior SWE-Bench (Senior SWE-Bench) A Snorkel AI benchmark of senior-level software engineering tasks emphasizing under-specified feature work, bug/performance investigation, and taste-based correctness. - **Year**: 2026 - **Tasks**: Senior-level repository tasks - **Format**: Agentic software-engineering evaluation - **Difficulty**: Professional senior engineering ### VulcanBench v3 (VulcanBench v3) An open software-engineering benchmark built from real merged post-cutoff pull requests across Python, Rust, TypeScript, JavaScript, and Go repositories. - **Year**: 2026 - **Tasks**: 23 post-cutoff repository tasks in the v3 report - **Format**: Pass@1 with low, medium, and high effort - **Difficulty**: Professional multi-file software engineering ### OpenHarmony Bench (OpenHarmony Bench v1.0) An app-level coding benchmark that asks DevEco Code configurations to implement observable behavior in buildable OpenHarmony ArkTS applications. - **Year**: 2026 - **Tasks**: 153 app-development and bug-fix tasks - **Format**: Task completion through DevEco Code - **Difficulty**: End-to-end OpenHarmony application development ### VulcanBench CII v1 (VulcanBench Coding Intelligence Index v1) A post-cutoff software-engineering benchmark with hidden functional tests and regression guards, reported for vendor coding-agent harnesses. - **Year**: 2026 - **Tasks**: 38 validated post-cutoff repository tasks - **Format**: Pass@1 with vendor coding-agent harnesses - **Difficulty**: Mid-band frontier software engineering ### FrontierCode 1.1 Main (FrontierCode 1.1 Main) Cognition's 100-task software-engineering benchmark for whether coding agents produce mergeable, production-quality pull requests, scored for correctness, tests, scope, style, and maintainability through maintainer-authored rubrics. - **Year**: 2026 - **Tasks**: 100 private Main tasks (150 in Extended) - **Format**: Repository task completion with maintainer rubrics - **Difficulty**: Frontier coding-agent quality ### FrontierCode 1.1 Extended (FrontierCode 1.1 Extended) Cognition's 150-task Extended subset of the FrontierCode 1.1 software-engineering benchmark. - **Year**: 2026 - **Tasks**: 150 private software-engineering tasks - **Format**: Repository task completion with maintainer rubrics - **Difficulty**: Frontier coding-agent quality ### IDE-Bench (IDE-Bench) An 80-task software-engineering benchmark across eight repositories that tests whether autonomous IDE agents can explore, edit, run, and verify code changes end to end. - **Year**: 2026 - **Tasks**: 80 tasks across 8 repositories - **Format**: Autonomous IDE-agent task completion (pass@1) - **Difficulty**: End-to-end software engineering ### App-Bench (App-Bench) A six-task full-stack web-app benchmark that measures how much required functionality an AI builder or coding assistant delivers from one prompt without human code edits. - **Year**: 2025 - **Tasks**: 6 full-stack app-building tasks - **Format**: Best-of-three one-shot feature completion - **Difficulty**: Production-style full-stack application generation ### SWE Multilingual (SWE Multilingual) A multilingual software-engineering benchmark for real-world code issue resolution across multiple programming languages. - **Year**: 2026 - **Tasks**: Multilingual software-engineering tasks - **Format**: Repository task completion - **Difficulty**: Professional software engineering ### SWE Multimodal (SWE-bench Multimodal) A multimodal variant of SWE-bench that adds visual context such as screenshots and design mockups to software engineering issue descriptions. - **Year**: 2025 - **Tasks**: Multimodal software engineering tasks - **Format**: Code patch generation with visual context - **Difficulty**: Frontier multimodal coding ### CursorBench (CursorBench) Cursor's first-party benchmark for ambiguous, multi-file coding-agent tasks drawn from real Cursor sessions, currently on the 4.0 task set. - **Year**: 2026 - **Tasks**: Long-horizon multi-file agentic coding tasks - **Format**: Cursor agent-loop evaluation - **Difficulty**: Professional agentic software engineering ### Multi-SWE Bench (Multi-SWE Bench) A multi-language software-engineering benchmark that measures repository-level bug fixing and implementation across more than one programming ecosystem. - **Year**: 2026 - **Tasks**: Multi-language repo tasks - **Format**: Repository task completion - **Difficulty**: Professional software engineering ### VIBE-Pro (VIBE-Pro) A repo-level code generation and full-project delivery benchmark spanning web, mobile, and simulation-style implementation tasks. - **Year**: 2026 - **Tasks**: Full project delivery tasks - **Format**: Repository-level implementation benchmark - **Difficulty**: End-to-end software delivery ### Vibe Code Bench (Vibe Code Bench v1.1) Vals.ai benchmark for evaluating whether models can build complete web applications from natural language specifications in a production-like development environment. - **Year**: 2026 - **Tasks**: End-to-end web application builds - **Format**: Full-stack app implementation benchmark - **Difficulty**: End-to-end software delivery ### ProgramBench (ProgramBench: Can Language Models Rebuild Programs From Scratch?) A cleanroom software-engineering benchmark where agents receive only a compiled executable and documentation, then must architect and implement a complete codebase that reproduces the original program's behavior. - **Year**: 2026 - **Tasks**: 200 program reconstruction tasks - **Format**: Cleanroom executable reimplementation - **Difficulty**: Full-repository software architecture ### PostTrain Bench (PostTrain Bench) A software-engineering benchmark for post-training infrastructure and implementation tasks, evaluated through the official Harbor implementation. - **Year**: 2026 - **Tasks**: Post-training software-engineering tasks - **Format**: Harbor agent evaluation - **Difficulty**: Frontier software engineering ### FrontierSWE (FrontierSWE) An ultra-long-horizon software-engineering benchmark with open-ended implementation, performance, and research tasks designed to challenge frontier coding agents. - **Year**: 2026 - **Tasks**: 17 ultra-long-horizon engineering and research tasks - **Format**: Mean@5, best@5, average rank, and dominance - **Difficulty**: Ultra-long-horizon frontier software engineering ### FrontierSWE v2 (FrontierSWE v2) A 34-task expansion of FrontierSWE for ultra-long-horizon engineering and research work that remains far from saturation. - **Year**: 2026 - **Tasks**: 34 ultra-long-horizon engineering and research tasks - **Format**: Five-trial mean task score (Mean@5), 0-100 - **Difficulty**: Ultra-long-horizon frontier software engineering ### SWE-Atlas Codebase QnA (SWE-Atlas Codebase QnA) A code-comprehension benchmark covering production repositories across Go, Python, C, and TypeScript. - **Year**: 2026 - **Tasks**: 124 codebase questions across 11 repositories - **Format**: Mean pass@1 - **Difficulty**: Production codebase comprehension ### Bug Hunt Bench (Bug Hunt Bench) A blind-graded coding-agent benchmark with 105 planted bugs across two production TypeScript repositories. - **Year**: 2026 - **Tasks**: 105 planted bugs across two production TypeScript repositories - **Format**: Strict planted bugs fixed - **Difficulty**: Blind production-repository bug finding and repair ### Kimi Code Bench v2 (Kimi Code Bench v2) A Moonshot AI internal coding-agent benchmark for realistic software-engineering tasks across mainstream programming languages and production technology stacks. - **Year**: 2026 - **Tasks**: Realistic coding-agent tasks - **Format**: Coding-agent pass rate - **Difficulty**: Production software engineering ### MLS-Bench Lite (MLS-Bench Lite) A 30-task subset of MLS-Bench that evaluates whether AI systems can invent generalizable and scalable machine-learning methods. - **Year**: 2026 - **Tasks**: 30 machine-learning research tasks - **Format**: Agentic ML task evaluation - **Difficulty**: ML research and systems engineering ### PaperBench (PaperBench) A research-reproduction benchmark that asks agents to recreate the contributions of AI papers from the paper alone. - **Year**: 2026 - **Tasks**: AI research-paper reproduction - **Format**: Long-horizon agent evaluation - **Difficulty**: Frontier autonomous research and engineering ### QwenReactBench (QwenReactBench) Qwen's internal benchmark for building and rendering bilingual React projects across seven categories. - **Year**: 2026 - **Tasks**: Bilingual React project construction - **Format**: Bradley-Terry/Elo rating - **Difficulty**: Production frontend development ### NL2Repo (NL2Repo) A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes. - **Year**: 2026 - **Tasks**: Natural language to repository tasks - **Format**: Repository understanding benchmark - **Difficulty**: System-level software comprehension ### DSBench-FullStack (DeepSeek DSBench FullStack) DeepSeek's internal full-stack coding-agent benchmark. - **Year**: 2026 - **Tasks**: Internal full-stack coding-agent tasks - **Format**: Provider-reported score - **Difficulty**: Full-stack software engineering ### DSBench-Hard (DeepSeek DSBench Hard) DeepSeek's internal hard coding-agent benchmark. - **Year**: 2026 - **Tasks**: Internal hard coding-agent tasks - **Format**: Provider-reported score - **Difficulty**: Advanced coding-agent challenges ### React Native Evals (React Native Evals) An open benchmark for AI coding agents on real-world React Native implementation tasks, emphasizing working app behavior, recommended architecture choices, and strict constraint adherence. - **Year**: 2026 - **Tasks**: React Native app implementation tasks - **Format**: Framework-specific app development evaluation - **Difficulty**: Production mobile app engineering ### ReactBench (ReactBench v1) A coding-agent benchmark for realistic React work, with rubrics that check production concerns such as performance, accessibility, correctness, and code quality. - **Year**: 2026 - **Tasks**: 51 production React tasks - **Format**: Pass@1 weighted rubric score - **Difficulty**: Production frontend engineering ### KernelBench (KernelBench Hard H100) An agentic GPU-kernel benchmark that measures how much of the hardware roofline a model's correct, audit-clean kernels reach on six demanding CUDA and Triton problems. - **Year**: 2026 - **Tasks**: 6 GPU-kernel optimization problems - **Format**: Mean peak fraction of hardware roofline over valid cells - **Difficulty**: Agentic GPU systems engineering ### Next.js Evals (AI Agent Evaluations for Next.js) A Vercel benchmark for AI coding agents on Next.js code generation and migration tasks, reporting pass@4, average execution time, and results with bundled documentation through AGENTS.md. - **Year**: 2026 - **Tasks**: 31 current Next.js code generation and migration tasks - **Format**: Success rate (pass@4), with and without bundled documentation - **Difficulty**: Framework-specific web application engineering ### Spider 2.0-Lite (Spider 2.0-Lite) A text-to-SQL benchmark over realistic warehouse-scale schemas, reported by Interfaze for model comparison. - **Year**: 2024 - **Tasks**: Text-to-SQL queries - **Format**: Execution accuracy - **Difficulty**: Enterprise text-to-SQL ### SciCode (Scientific Code Benchmark) SciCode evaluates language models on generating code for realistic scientific research problems across 16 subfields of physics, math, chemistry, biology, and material science. Problems decompose into 338 subproblems requiring domain knowledge recall, scientific reasoning, and precise code synthesis. Based on real scripts from published research. - **Year**: 2024 - **Tasks**: 80 ### AA Coding Index (Artificial Analysis Coding Index) A display-only Artificial Analysis coding index. - **Year**: 2026 - **Tasks**: Cross-benchmark coding index - **Format**: Aggregated model score - **Difficulty**: Display-only external reference ### AA Coding Agents (Artificial Analysis Coding Agent Index) A display-only Artificial Analysis leaderboard for coding-agent systems, combining agent harnesses, host models, and execution settings across software-engineering benchmarks. - **Year**: 2026 - **Tasks**: Composite over DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnA - **Format**: Average pass@1 index - **Difficulty**: Real-world coding-agent workflows ### AA-SciCode (Artificial Analysis SciCode) An Artificial Analysis SciCode score. - **Year**: 2026 - **Tasks**: Scientific coding subproblems - **Format**: Task success rate - **Difficulty**: Scientific programming ### Terminal-Bench Hard (Terminal-Bench Hard) A display-only Artificial Analysis coding metric for agentic coding and terminal use on a harder Terminal-Bench slice. - **Year**: 2026 - **Tasks**: Agentic coding and terminal tasks - **Format**: Task success rate - **Difficulty**: Professional software engineering ### VIBE V2 (VIBE V2) A display-only MiniMax provider benchmark for end-to-end coding-agent and product-building tasks. - **Year**: 2026 - **Tasks**: End-to-end coding-agent tasks - **Format**: Task success rate - **Difficulty**: Frontier coding-agent workflows ### SVG-Bench (SVG-Bench) A display-only provider benchmark for generating or manipulating SVG outputs from natural-language requirements. - **Year**: 2026 - **Tasks**: SVG generation and editing tasks - **Format**: Task success rate - **Difficulty**: Visual coding and structured graphics generation ### KernelBench Hard (KernelBench Hard) A display-only benchmark for difficult GPU kernel implementation and optimization tasks. - **Year**: 2026 - **Tasks**: Hard GPU kernel coding tasks - **Format**: Task success rate - **Difficulty**: Specialized systems programming ### GameDevBench (GameDevBench) Evaluates coding agents on 333 multimodal game-development tasks in Godot, spanning 2D graphics, 3D graphics, user interfaces, and gameplay logic. - **Year**: 2026 - **Tasks**: 333 tasks from 88 tutorials - **Format**: Pass@1 on the full task set with 95% confidence intervals - **Difficulty**: Multimodal game development in Godot 4.4.1 ### EdgeBench (EdgeBench) A systems and software-engineering benchmark from ByteDance Seed that evaluates agents on long-horizon edge tasks using time-budgeted learning curves rather than a single static pass rate. - **Year**: 2026 - **Tasks**: Systems and software-engineering tasks - **Format**: Time-budgeted agent learning curves - **Difficulty**: Long-horizon engineering ### LiveCodeBench (Vals) (LiveCodeBench, Vals AI run) Vals AI’s independent implementation of the LiveCodeBench code-generation benchmark, run under one fixed harness across the models it tracks. - **Year**: 2026 - **Tasks**: Competitive programming problems (easy, medium, hard) - **Format**: Pass@1 accuracy - **Difficulty**: Frontier coding ### SWE-bench (Vals) (SWE-bench, Vals AI run) Vals AI’s independent run of the public SWE-bench issue set, reported by human time-to-fix bucket. Vals removed SWE-bench Verified from its index as saturated on 2026-05-04; this board is the standalone SWE-bench run. - **Year**: 2026 - **Tasks**: Real repository issues by human time bucket - **Format**: Resolved rate - **Difficulty**: Frontier coding agents ### EEBench (EEBench V1 core corpus) An electrical-engineering agent benchmark maintained by atopile. - **Year**: 2026 - **Tasks**: Electrical-engineering design tasks - **Format**: Agent task score - **Difficulty**: Professional electrical engineering ### CADGenBench Generation (CADGenBench Generation) Hugging Face's open benchmark for generating valid and geometrically correct 3D mechanical parts. - **Year**: 2026 - **Tasks**: 49 CAD generation samples - **Format**: Validity-gated CAD score - **Difficulty**: Professional mechanical CAD ### DeepSWE (DeepSWE) A long-horizon software engineering benchmark from Datacurve for measuring frontier coding agents on original tasks drawn from active open-source repositories. - **Year**: 2026 - **Tasks**: 113 software engineering tasks across 91 repositories and 5 languages - **Format**: Pass@1 with confidence interval, cost, time, and token metadata - **Difficulty**: Long-horizon software engineering ### SWE-Bench Pro V2 Full (Scale Labs SWE-Bench Pro V2 Full) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 15 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### SWE-Bench Pro V2 HARD (Scale Labs SWE-Bench Pro V2 HARD) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 15 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### SWE Atlas – Codebase QnA (Scale Labs SWE Atlas – Codebase QnA) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 24 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### SWE Atlas – Refactoring (Scale Labs SWE Atlas – Refactoring) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 17 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### SWE Atlas – Test Writing (Scale Labs SWE Atlas – Test Writing) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 24 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### SWE-bench Multimodal (Vals) (Vals SWE-bench Multimodal evaluation) JavaScript issue resolution where bug reports depend on screenshots, evaluated by Vals AI with no agent internet access. - **Year**: 2026 - **Tasks**: 480 JavaScript issues with visual bug reports - **Format**: Resolved issue percentage - **Difficulty**: Visual software issue resolution ### BridgeBench (BridgeBench) BridgeBench rates 25 models across nine software-building axes, including reasoning, front end, back end, security, speed, and cost. Its overall rating is the equal-weight mean of those axes. - **Year**: 2026 - **Tasks**: No public task count - **Format**: Editorial Elo-style rating - **Difficulty**: Software-building assessments ### SWE-Marathon (SWE-Marathon) A long-horizon software engineering benchmark from Abundant AI with multi-hour tasks spanning library reproductions, full-stack product clones, and ML engineering. - **Year**: 2026 - **Tasks**: 20 multi-hour software engineering tasks - **Format**: Task resolution and trajectory review - **Difficulty**: Ultra-long-horizon software engineering ### GBA-Eval (GBA-Eval) An agentic coding benchmark that asks models to build a Game Boy Advance emulator from scratch and grades emulator behavior against procedural, audio, and gameplay tests. - **Year**: 2026 - **Tasks**: 27 emulator test cases - **Format**: Overall emulator score - **Difficulty**: Long-horizon systems programming ### WeirdML (WeirdML v2) A machine-learning engineering benchmark that tests whether LLMs can train models on novel datasets, write PyTorch code, and improve through iterative feedback. - **Year**: 2026 - **Tasks**: 17 novel ML engineering tasks - **Format**: Average accuracy across tasks - **Difficulty**: Novel dataset modeling and iterative debugging ### Code Migration (Vals Code Migration) Can language models reimplement real-world programs in another language? - **Year**: 2026 - **Tasks**: Real-world program reimplementation in another language - **Format**: Accuracy score - **Difficulty**: Production code migration ### IOI (Vals IOI) Based on the International Olympiad in Informatics - **Year**: 2026 - **Tasks**: International Olympiad in Informatics-style programming tasks - **Format**: Accuracy score - **Difficulty**: Olympiad programming ### IOI v1 (Vals, archived) (Vals IOI v1 archived leaderboard) Archived Vals AI results on 2024 and 2025 International Olympiad in Informatics programming tasks. - **Year**: 2026 - **Tasks**: IOI 2024 and 2025 programming tasks - **Format**: Accuracy score - **Difficulty**: Olympiad programming ### ProgramBench (Vals ProgramBench) Can language models rebuild programs from scratch? - **Year**: 2026 - **Tasks**: Program reconstruction tasks - **Format**: Accuracy score - **Difficulty**: Cleanroom software engineering ### Vibe Code Bench 1-100 (Vals Vibe Code Bench 1-100) Can a model extend one working web application across a long sequence of dependent requests? - **Year**: 2026 - **Tasks**: Long chains of dependent web-application change requests - **Format**: Accuracy score - **Difficulty**: Long-horizon software delivery ### APEX-SWE (Terminus-2) (Mercor) (APEX-SWE (Terminus-2) — Mercor evaluation) Software integration and production-failure diagnosis with the Terminus-2 harness. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: Integration and observability engineering tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### APEX-SWE (Inspect) (Mercor) (APEX-SWE (Inspect) — Mercor evaluation) Software integration and production-failure diagnosis with the Inspect harness. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: Integration and observability engineering tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### SWE Atlas - Codebase QnA Extended (Mercor) (SWE Atlas - Codebase QnA Extended — Mercor evaluation) Answering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 50 Mercor tasks; 124 tasks in the separately reported original set - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### SWE-bench Verified Extended (Mercor) (SWE-bench Verified Extended — Mercor evaluation) Editing repositories to resolve verified GitHub issues. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 50 Mercor tasks; 500 tasks in the separately reported original set - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### SWE Atlas - Codebase QnA (Mercor) (SWE Atlas - Codebase QnA — Mercor evaluation) Answering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 124 source-reported tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### SciCode (Mercor) (SciCode — Mercor evaluation) Solving scientific programming subproblems with executable checks. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 65 source-reported tasks - **Format**: Mean Score - **Difficulty**: Source-specific evaluation ### CVDP (Mercor) (CVDP — Mercor evaluation) Hardware design, verification, and comprehension tasks. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 517 source-reported tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### DeepSWE v1.1 (Mercor) (DeepSWE v1.1 — Mercor evaluation) Resolving software issues on the DeepSWE v1.1 task set. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 113 source-reported tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### ProgramBench (Mercor) (ProgramBench — Mercor evaluation) Reconstructing a program from its executable behavior and documentation. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 100 source-reported tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### Web Clone Bench (Web Clone Bench v1) Coding agents recreate nine reference web apps from an empty directory. Plato checks visual and functional behavior through end-to-end workflows. - **Year**: 2026 - **Tasks**: 9 reference web apps - **Format**: Published full score (%) - **Difficulty**: Long-running web application recreation ## Reasoning Benchmarks ### STEMBench (STEMBench120: six-domain scientific research progress) Scores how far a model advances 120 open research tasks across biology, chemistry, physics, astronomy, Earth science and engineering on a 0 to 4 progress scale, crediting supported partial work and honest statements of what remains unresolved. - **Year**: 2026 - **Tasks**: 120 research tasks, 20 in each of six STEM domains - **Format**: Audited progress level 0 to 4 per task, refined to a fractional score; complete-run mean across all tasks - **Difficulty**: Open research targets; no run has reached a full-target success ### GraphWalks BFS 256K–1M (GraphWalks BFS 256K–1M) Breadth-first-search graph traversal on 200 problems with context lengths from 256K to 1M tokens. - **Year**: 2026 - **Tasks**: Breadth-first-search graph traversal on 200 problems with context lengths from 256K to 1M tokens. - **Format**: F1 - **Difficulty**: Frontier agent evaluation ### ARC-AGI-1 (ARC-AGI-1 Semi-Private Evaluation) ARC Prize fluid-intelligence benchmark using novel visual grid transformations. - **Year**: 2026 - **Tasks**: Semi-private ARC-AGI-1 evaluation set - **Format**: Verified accuracy - **Difficulty**: Abstract visual reasoning ### MuSR (Testing the Limits of Chain-of-thought with Multistep Soft Reasoning) A dataset for evaluating language models on multistep soft reasoning tasks specified in natural language narratives. Tests the ability to perform complex, structured reasoning. - **Year**: 2023 - **Tasks**: Multi-step reasoning - **Format**: Narrative-based reasoning - **Difficulty**: Complex reasoning tasks ### BBH (BIG-Bench Hard) A suite of 23 challenging tasks from the BIG-Bench collaborative benchmark where prior language models failed to exceed average human performance, even with chain-of-thought prompting. - **Year**: 2022 - **Tasks**: 23 tasks - **Format**: Mixed reasoning tasks - **Difficulty**: Advanced reasoning ### DROP (Discrete Reasoning Over Paragraphs) A reading-comprehension benchmark requiring discrete reasoning over paragraphs, reported in DeepSeek-V4 base-model evaluations. - **Year**: 2026 - **Tasks**: Paragraph reasoning questions - **Format**: F1 - **Difficulty**: Reading and numerical reasoning ### HellaSwag (HellaSwag) A commonsense natural-language inference benchmark reported in DeepSeek-V4 base-model evaluations. - **Year**: 2026 - **Tasks**: Commonsense completion questions - **Format**: Exact match - **Difficulty**: Commonsense reasoning ### WinoGrande (WinoGrande) A commonsense coreference benchmark reported in DeepSeek-V4 base-model evaluations. - **Year**: 2026 - **Tasks**: Coreference resolution questions - **Format**: Exact match - **Difficulty**: Commonsense reasoning ### CLUEWSC (CLUEWSC) A Chinese Winograd Schema Challenge benchmark reported in DeepSeek-V4 base-model evaluations. - **Year**: 2026 - **Tasks**: Chinese coreference questions - **Format**: Exact match - **Difficulty**: Chinese commonsense reasoning ### LisanBench (LisanBench) A word-chain reasoning benchmark that tests planning, recall, constraint following, and vocabulary depth by asking models to extend non-repeating edit-distance-1 chains. - **Year**: 2026 - **Tasks**: 50 starting words × 3 trials - **Format**: Difficulty-weighted word-chain reasoning - **Difficulty**: Open-ended lexical planning ### Conceptual Reasoning (Conceptual Reasoning Benchmark) Tests whether model judgments rank argumentative critiques in the same order as expert human ratings across philosophy, AI alignment, and other concept-heavy texts. - **Tasks**: 224 texts and 608 within-text critique pairs - **Format**: Average pairwise-ranking loss against expert ratings - **Difficulty**: Fuzzy, expert-rated argumentative reasoning ### Pencil Puzzle Bench (Pencil Puzzle Bench) A multi-step verifiable reasoning benchmark that evaluates whether models can solve pencil puzzles with unique solutions. - **Year**: 2026 - **Tasks**: 300 evaluation puzzles - **Format**: Direct and agentic puzzle solve rate - **Difficulty**: Multi-step verifiable reasoning ### LongBench v2 (LongBench v2) A long-context benchmark that measures whether models can actually use extended context windows for reasoning and retrieval. - **Year**: 2025 - **Tasks**: Long-context tasks - **Format**: Extended-context retrieval and reasoning - **Difficulty**: Hard long-context ### MRCRv2 (MRCRv2) A long-context benchmark for memory, retrieval, and multi-round coherence over large contexts. - **Year**: 2025 - **Tasks**: Long-context retrieval - **Format**: Multi-round long-context evaluation - **Difficulty**: Hard long-context ### MRCR v2 64K-128K (OpenAI MRCR v2 8-needle 64K-128K) MRCR v2 slice focused on long-context retrieval at 64K-128K lengths. - **Year**: 2026 - **Tasks**: 8-needle retrieval tasks - **Format**: Long-context retrieval - **Difficulty**: Long-context reasoning ### MRCR v2 128K-256K (OpenAI MRCR v2 8-needle 128K-256K) MRCR v2 slice focused on very long contexts at 128K-256K lengths. - **Year**: 2026 - **Tasks**: 8-needle retrieval tasks - **Format**: Very-long-context retrieval - **Difficulty**: Very long-context reasoning ### MRCR v2 256K-512K (OpenAI MRCR v2 8-needle 256K-512K) MRCR v2 slice focused on retrieval across 256K-512K-token contexts. - **Year**: 2026 - **Tasks**: 100 eight-needle retrieval examples - **Format**: Mean sequence-matcher ratio - **Difficulty**: Very long-context retrieval ### MRCR v2 512K-1M (OpenAI MRCR v2 8-needle 512K-1M) MRCR v2 slice focused on retrieval across 512K-1M-token contexts. - **Year**: 2026 - **Tasks**: 100 eight-needle retrieval examples - **Format**: Mean sequence-matcher ratio - **Difficulty**: Million-token retrieval ### Graphwalks BFS 128K (Graphwalks BFS 0K-128K) Long-context graph traversal benchmark using breadth-first search tasks. - **Year**: 2026 - **Tasks**: Graph traversal tasks - **Format**: Long-context graph reasoning - **Difficulty**: Algorithmic long-context reasoning ### Graphwalks Parents 128K (Graphwalks parents 0-128K) Long-context benchmark for recovering parent relationships inside graph tasks. - **Year**: 2026 - **Tasks**: Graph parent-retrieval tasks - **Format**: Long-context graph reasoning - **Difficulty**: Algorithmic long-context reasoning ### MRCR 1M (MRCR 1M) A million-token MRCR long-context retrieval benchmark reported in DeepSeek-V4 model evaluations. - **Year**: 2026 - **Tasks**: Million-token retrieval - **Format**: Long-context retrieval MMR - **Difficulty**: Million-token long context ### CorpusQA 1M (CorpusQA 1M) A million-token CorpusQA long-context question-answering benchmark reported in DeepSeek-V4 model evaluations. - **Year**: 2026 - **Tasks**: Million-token corpus question answering - **Format**: Long-context QA accuracy - **Difficulty**: Million-token long context ### ARC-AGI-2 (Abstraction and Reasoning Corpus for AGI v2) ARC-AGI-2 tests whether a system can infer a transformation from example grids and apply it to a new grid. Compare accuracy with the evaluation split, attempt limit, and cost attached. - **Year**: 2025 - **Tasks**: Visual pattern completion and abstract reasoning - **Format**: Grid transformation puzzles with novel rules - **Difficulty**: Expert-level — hardest public reasoning benchmark ### ARC-AGI-3 (Abstraction and Reasoning Corpus for AGI v3) An interactive successor to ARC-AGI-2 that evaluates whether an AI agent can learn unfamiliar task mechanics through action and feedback. - **Year**: 2026 - **Tasks**: Interactive game-like tasks with hidden rules - **Format**: Agentic task completion under a capped evaluation budget - **Difficulty**: Frontier agentic reasoning ### GeneBench-Pro (GeneBench-Pro) A multistage statistical-reasoning benchmark for genomics and biological-data analysis agents. - **Year**: 2026 - **Tasks**: 129 genomics statistical-analysis workflows - **Format**: Eval-level pass rate across dependent analysis decisions - **Difficulty**: Long-horizon scientific reasoning ### AI-Needle (AI-Needle) A long-context retrieval benchmark that measures whether a model can recover relevant information embedded deep inside very long contexts. - **Year**: 2026 - **Tasks**: Long-context retrieval - **Format**: Needle-in-a-haystack recall - **Difficulty**: Long-context memory ### GPQA Diamond (GPQA Diamond) The hardest subset of GPQA featuring the most challenging graduate-level science questions. Sometimes reported separately from the standard GPQA benchmark. - **Year**: 2023 - **Tasks**: Expert-level science questions - **Format**: Multiple choice questions - **Difficulty**: Graduate-level scientific reasoning ### AA-LCR (Artificial Analysis Long Context Reasoning) An Artificial Analysis long-context reasoning evaluation. - **Year**: 2026 - **Tasks**: Long-context reasoning tasks - **Format**: Accuracy - **Difficulty**: Long-context reasoning ### MLCR-AA (Medical Long Context Reasoning (MLCR-AA)) An open benchmark from Wisedocs, run by Artificial Analysis, measuring how well models reason over long, fragmented medical records with multi-document synthesis. - **Year**: 2026 - **Tasks**: Long, fragmented medical-record reasoning - **Format**: Accuracy - **Difficulty**: Long-context medical reasoning ### CritPt (Critical Physics Tasks) An Artificial Analysis metric for research-level physics reasoning. - **Year**: 2026 - **Tasks**: Research-level physics questions - **Format**: Accuracy - **Difficulty**: Research-level physics reasoning ### BullshitBench v2 (BullshitBench v2) A benchmark that tests whether AI models challenge nonsensical, ill-posed, or logically flawed prompts instead of confidently generating incorrect answers. Measures the critical ability to push back on bad input. - **Year**: 2025 - **Tasks**: Nonsensical and flawed prompts across multiple domains - **Format**: Prompt challenge and refusal evaluation - **Difficulty**: Robustness and critical reasoning ### WildBench (WildBench) An automated evaluation framework using 1,000+ real-world user tasks covering reasoning, planning, coding, and creative writing. Highly correlated with Chatbot Arena human preference rankings. - **Year**: 2024 - **Tasks**: 1,024 real-world tasks - **Format**: Real-world task evaluation - **Difficulty**: Diverse real-world scenarios ### EnigmaEval (Scale Labs EnigmaEval) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 5 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### PropensityBench (Scale Labs PropensityBench) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 14 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### RULER (RULER context-length reports) Selected original RULER context sweeps and separate IBM Granite 4.2 reports. The table keeps each source cohort and context length explicit; 128K is the displayed score, and the results do not enter rankings. - **Year**: 2024 - **Tasks**: 13 tasks in the original author cohort; IBM task count undisclosed - **Format**: Reported score by context length ### NoLiMa (NoLiMa original-task reports) Selected author accuracy curves for long-context retrieval with minimal wording overlap. Original NoLiMa stays separate from Hard, and 64K/128K use fewer placements. The displayed score is 32K accuracy, outside overall and category rankings. - **Year**: 2025 - **Tasks**: 58 question-needle pairs; five haystacks; 26 placements through 32K - **Format**: Character-name answer accuracy by context length ### NoLiMa-Hard (NoLiMa-Hard selected reports) Selected reasoning-model accuracy curves on the ten hardest NoLiMa question-needle pairs. Exact API snapshots and effort remain undisclosed. The displayed score is 32K accuracy; these reports stay outside overall and category rankings. - **Year**: 2025 - **Tasks**: 10 hardest question-needle pairs - **Format**: Hard-subset character-name answer accuracy ### Long Context Reasoning (AA-LCR) Extended (Mercor) (Long Context Reasoning (AA-LCR) Extended — Mercor evaluation) Synthesizing information across long collections of documents. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 89 Mercor tasks; 100 tasks in the separately reported original set - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### Long Context Reasoning (AA-LCR v1.1) (Mercor) (Long Context Reasoning (AA-LCR v1.1) — Mercor evaluation) Synthesizing information across long collections of documents. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 100 source-reported tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ## Multimodal & Grounded Benchmarks ### Chartography (no tools) (Chartography without tools) Professional chart understanding across 100 specialized chart types with expert-set answer tolerances. - **Year**: 2026 - **Tasks**: 100 specialized chart types - **Format**: Accuracy without tools - **Difficulty**: Professional chart reasoning ### Chartography (tools) (Chartography with image and code tools) Professional chart understanding with a container, standard libraries, and image cropping. - **Year**: 2026 - **Tasks**: 100 specialized chart types - **Format**: Accuracy with tools - **Difficulty**: Professional chart reasoning ### BenchCAD Vision2Code (no tools) (BenchCAD Vision2Code voxel IoU without tools) Generates CadQuery code from multi-view renders and scores geometric similarity by voxel intersection-over-union. - **Year**: 2026 - **Tasks**: 1,000-file Vision2Code subset - **Format**: Voxel IoU - **Difficulty**: Programmatic CAD generation ### BenchCAD Vision2Code (tools) (BenchCAD Vision2Code voxel IoU with tools) Generates CadQuery code from multi-view renders with image inspection and code-execution tools. - **Year**: 2026 - **Tasks**: 1,000-file Vision2Code subset - **Format**: Voxel IoU with tools - **Difficulty**: Programmatic CAD generation ### Biomedical image analysis (Anthropic biomedical-image-analysis evaluation) Quantitative biomedical image analysis across segmentation, detection, measurement, classification, registration, reconstruction, and tracking tasks. - **Year**: 2026 - **Tasks**: 91 cases across 31 biomedical imaging tasks - **Format**: Normalized task metric - **Difficulty**: Biomedical image analysis ### GDP.pdf (no tools) (GDP.pdf mean criteria pass rate without tools) Professional document understanding over 100 real-world PDFs from ten domains. - **Year**: 2026 - **Tasks**: 100 professional document prompts - **Format**: Mean criteria pass rate - **Difficulty**: Professional document reasoning ### GDP.pdf (tools) (GDP.pdf mean criteria pass rate with tools) Professional document understanding with a container, standard libraries, and image cropping. - **Year**: 2026 - **Tasks**: 100 professional document prompts - **Format**: Mean criteria pass rate with tools - **Difficulty**: Professional document reasoning ### OfficeQA (OfficeQA) Grounded numerical reasoning over a corpus of historical U.S. Treasury Bulletin documents. - **Year**: 2026 - **Tasks**: Historical Treasury Bulletin questions - **Format**: Agentic grounded QA accuracy - **Difficulty**: Professional document reasoning ### MMMU (Massive Multi-discipline Multimodal Understanding) A broad multimodal reasoning benchmark spanning charts, diagrams, tables, and academic visual question answering. - **Year**: 2024 - **Tasks**: Multimodal academic reasoning - **Format**: Image + text question answering - **Difficulty**: Frontier multimodal ### MMMU-Pro (Massive Multi-discipline Multimodal Understanding Pro) A harder multimodal benchmark for frontier models that combines text with images, diagrams, charts, and academic visual reasoning tasks. - **Year**: 2024 - **Tasks**: Multimodal academic reasoning - **Format**: Image + text question answering - **Difficulty**: Frontier multimodal ### OCRBench V2 (OCRBench V2) A native OCR benchmark for reading text from images across multilingual scripts, low-quality scans, handwriting, structured layouts, charts, and screenshots. - **Year**: 2025 - **Tasks**: Image OCR tasks - **Format**: Accuracy - **Difficulty**: Native visual text understanding ### olmOCR (olmOCR-Bench) An end-to-end document understanding benchmark over long, layout-rich PDFs with tables, equations, headers, footnotes, and multi-column flows. - **Year**: 2025 - **Tasks**: Layout-rich PDF understanding - **Format**: Mean accuracy - **Difficulty**: Complex document processing ### VoxPopuli WER (VoxPopuli-Cleaned-AA Word Error Rate) A speech-recognition benchmark on the cleaned Artificial Analysis VoxPopuli subset, reported as word error rate where lower is better. - **Year**: 2026 - **Tasks**: Speech-to-text transcription - **Format**: Word error rate - **Difficulty**: Audio speech recognition ### Design Arena Website (Design Arena Website Elo) A display-only Design Arena website-generation Elo score surfaced on OpenRouter model benchmark pages. - **Year**: 2026 - **Tasks**: Website generation comparisons - **Format**: Elo - **Difficulty**: Design and website generation ### OfficeQA Pro (OfficeQA Pro) A benchmark for grounded reasoning over office-style documents, spreadsheets, charts, and business artifacts. - **Year**: 2026 - **Tasks**: Document and spreadsheet tasks - **Format**: Grounded QA over office artifacts - **Difficulty**: Enterprise grounded reasoning ### MathVision w/ Python (MathVision with Python) A tool-augmented MathVision variant that permits Python during visual mathematics reasoning. - **Year**: 2026 - **Tasks**: Visual mathematics problems with Python - **Format**: Image and mathematics reasoning with tools - **Difficulty**: Advanced multimodal mathematics ### BabyVision w/ Python (BabyVision with Python) A Python-assisted BabyVision evaluation for fine-grained visual perception and grounded reasoning. - **Year**: 2026 - **Tasks**: Visual perception tasks with Python - **Format**: Tool-augmented multimodal score - **Difficulty**: Fine-grained visual perception ### ZeroBench w/ Python (ZeroBench_main with Python) A Python-assisted ZeroBench_main evaluation reported as pass@5. - **Year**: 2026 - **Tasks**: Visual reasoning questions with Python - **Format**: Pass@5 - **Difficulty**: Tool-augmented visual reasoning ### WorldVQA ForceAnswer (WorldVQA ForceAnswer) A forced-answer WorldVQA variant for atomic visual world knowledge. - **Year**: 2026 - **Tasks**: Atomic visual world-knowledge questions - **Format**: Forced-answer visual QA - **Difficulty**: Fine-grained visual knowledge ### OmniDocBench (OmniDocBench) A document-understanding benchmark for parsing and reasoning over complex document layouts. - **Year**: 2026 - **Tasks**: Complex document-understanding tasks - **Format**: Document-understanding score - **Difficulty**: Grounded document reasoning ### PerceptionBench (PerceptionBench (Internal)) Moonshot AI's internal benchmark for atomic visual perception capabilities. - **Year**: 2026 - **Tasks**: Internal atomic visual-perception tasks - **Format**: Internal evaluation score - **Difficulty**: Fine-grained visual perception ### MMMU-Pro w/ Python (MMMU-Pro with Python) Tool-augmented MMMU-Pro variant that allows Python assistance during multimodal reasoning. - **Year**: 2026 - **Tasks**: Multimodal academic reasoning - **Format**: Image + text question answering with Python - **Difficulty**: Frontier multimodal ### OmniDocBench 1.5 (OmniDocBench 1.5) A document understanding benchmark used in frontier-model comparison tables to measure extraction and grounded reasoning quality on complex documents. - **Year**: 2026 - **Tasks**: Document understanding tasks - **Format**: Document understanding benchmark - **Difficulty**: Grounded document reasoning ### Liquid Extract JSON Validity (Liquid image-to-JSON extraction JSON validity) A display-only Liquid AI extraction metric measuring the share of image-to-JSON outputs that parse as strict JSON. - **Year**: 2026 - **Tasks**: Image-to-JSON extraction - **Format**: Strict JSON parseability rate - **Difficulty**: Structured visual extraction ### Liquid Extract F1 (Liquid image-to-JSON extraction schema consistency F1) A display-only Liquid AI extraction metric measuring field-name agreement between requested schema fields and extracted JSON fields. - **Year**: 2026 - **Tasks**: Image-to-JSON extraction - **Format**: Schema field F1 - **Difficulty**: Structured visual extraction ### Liquid Extract VLM Judge (Liquid image-to-JSON extraction VLM judge score) A display-only Liquid AI extraction metric measuring judged agreement between extracted values and the source image. - **Year**: 2026 - **Tasks**: Image-to-JSON extraction - **Format**: VLM-judged extraction accuracy - **Difficulty**: Structured visual extraction ### RealWorldQA (RealWorldQA) A grounded visual QA benchmark focused on answering practical questions about real-world images and scenes. - **Year**: 2026 - **Tasks**: Real-world visual question answering - **Format**: Image-grounded QA - **Difficulty**: General visual reasoning ### Video-MME (with subtitle) (Video-MME with subtitle) A video understanding benchmark that allows subtitle access when answering multimodal questions about videos. - **Year**: 2026 - **Tasks**: Video understanding - **Format**: Video QA with subtitle context - **Difficulty**: Multimodal video reasoning ### Video-MME (w/o subtitle) (Video-MME without subtitle) A stricter Video-MME setting that removes subtitle help and tests video understanding from visual and audio context alone. - **Year**: 2026 - **Tasks**: Video understanding - **Format**: Video QA without subtitle context - **Difficulty**: Multimodal video reasoning ### Video-MME (Video-MME) A comprehensive benchmark for multimodal large language models on video understanding, covering temporal reasoning, perception, and question answering over videos. - **Year**: 2024 - **Tasks**: Video understanding - **Format**: Video QA and analysis - **Difficulty**: Broad multimodal video reasoning ### MathVision (MathVision) A visual mathematics benchmark that tests whether a model can solve math problems grounded in diagrams, equations, figures, and other visual inputs. - **Year**: 2026 - **Tasks**: Visually grounded math problems - **Format**: Image + math reasoning - **Difficulty**: Advanced multimodal mathematics ### We-Math (We-Math) A multimodal math benchmark for visually grounded mathematical reasoning and answer generation. - **Year**: 2026 - **Tasks**: Visually grounded math problems - **Format**: Multimodal mathematical reasoning - **Difficulty**: Advanced multimodal mathematics ### DynaMath (DynaMath) A multimodal benchmark for dynamic mathematical reasoning over visual and structured inputs. - **Year**: 2026 - **Tasks**: Dynamic visual math problems - **Format**: Multimodal mathematical reasoning - **Difficulty**: Advanced multimodal mathematics ### MStar (MStar) A general visual question-answering benchmark used in provider tables for real-image reasoning quality. - **Year**: 2026 - **Tasks**: Real-image visual QA - **Format**: Image-grounded QA - **Difficulty**: General visual reasoning ### ChatCVQA (ChatCVQA) A conversational visual QA benchmark that tests multi-turn grounded answering over images and documents. - **Year**: 2026 - **Tasks**: Conversational visual QA - **Format**: Multi-turn image-grounded QA - **Difficulty**: Conversational multimodal reasoning ### MMLongBench-Doc (MMLongBench-Doc) A long-document multimodal benchmark for grounded reasoning over extended document contexts. - **Year**: 2026 - **Tasks**: Long document understanding - **Format**: Document-grounded reasoning - **Difficulty**: Long-context document reasoning ### CC-OCR (CC-OCR) An OCR-focused benchmark for reading and extracting text from visually complex documents and images. - **Year**: 2026 - **Tasks**: Optical character recognition - **Format**: Text extraction from images and documents - **Difficulty**: Document reading ### AI2D_TEST (AI2D test split) A diagram understanding benchmark focused on scientific and educational visual question answering. - **Year**: 2026 - **Tasks**: Diagram understanding - **Format**: Diagram-grounded QA - **Difficulty**: Structured visual reasoning ### CountBench (CountBench) A visual counting benchmark that tests whether a model can count objects and entities reliably in complex scenes. - **Year**: 2026 - **Tasks**: Visual counting tasks - **Format**: Image-grounded counting - **Difficulty**: Fine-grained visual perception ### RefCOCO (avg) (RefCOCO average) A referring-expression grounding benchmark averaged across RefCOCO variants to test whether a model can localize described objects correctly. - **Year**: 2026 - **Tasks**: Referring-expression grounding - **Format**: Grounded visual localization - **Difficulty**: Fine-grained visual grounding ### ODINW13 (ODINW13) A visual detection and grounding benchmark slice used to compare zero-shot object understanding across diverse domains. - **Year**: 2026 - **Tasks**: Out-of-distribution object understanding - **Format**: Detection and grounding - **Difficulty**: Robust visual grounding ### ERQA (ERQA) A grounded visual reasoning benchmark focused on evidence-based question answering over real images. - **Year**: 2026 - **Tasks**: Evidence-based visual QA - **Format**: Grounded image reasoning - **Difficulty**: Grounded multimodal reasoning ### VideoMMMU (VideoMMMU) A video extension of MMMU-style multimodal reasoning over expert questions grounded in temporal media. - **Year**: 2026 - **Tasks**: Video-grounded expert reasoning - **Format**: Video + text reasoning - **Difficulty**: Frontier multimodal video reasoning ### MLVU (M-Avg) (MLVU mean average) A multi-task video understanding benchmark averaged across MLVU categories. - **Year**: 2026 - **Tasks**: General video understanding - **Format**: Video QA and understanding - **Difficulty**: Broad multimodal video reasoning ### LVBench (LVBench) A long-video understanding benchmark for retrieving and reasoning over information distributed across extended video inputs. - **Year**: 2026 - **Tasks**: Long-form video question answering - **Format**: Long-video understanding score - **Difficulty**: Extended temporal reasoning ### MMVU (Multimodal Multi-disciplinary Video Understanding) A benchmark for evaluating multimodal models on video understanding tasks across multiple disciplines, emphasizing temporal reasoning and comprehension over video content. - **Year**: 2026 - **Tasks**: Video understanding - **Format**: Video reasoning benchmark - **Difficulty**: Multi-disciplinary multimodal video reasoning ### ScreenSpot Pro (ScreenSpot Pro) A GUI-grounding benchmark for 1,581 instructions in full-screen, high-resolution professional interfaces. It tests where a target is, not whether an agent can finish the surrounding workflow. - **Year**: 2025 - **Tasks**: 1,581 grounding instructions - **Format**: Static interface element localization - **Difficulty**: Professional GUI grounding ### TIR-Bench (TIR-Bench) A visual agent benchmark for interface reasoning and task execution over screenshots or software surfaces. - **Year**: 2026 - **Tasks**: Visual agent and interface reasoning - **Format**: Screenshot-grounded task reasoning - **Difficulty**: Computer-use visual reasoning ### GDPval-AA (GDPval-AA) An evaluation focused on professional domain expertise and task delivery quality in office-style knowledge work. - **Year**: 2026 - **Tasks**: Professional office delivery - **Format**: ELO-style office benchmark - **Difficulty**: Professional knowledge work ### MedXpertQA (MM) (MedXpertQA Multimodal) A multimodal medical multiple-choice benchmark covering clinical images such as X-rays, histology, and dermatology. - **Year**: 2026 - **Tasks**: 2,000 multimodal medical questions - **Format**: Medical visual MCQ - **Difficulty**: Clinical multimodal reasoning ### ZeroBench (ZeroBench) A multi-step visual reasoning benchmark with pass@5 reporting and optional tool use. - **Year**: 2026 - **Tasks**: 100 visual reasoning questions - **Format**: Multi-step visual reasoning - **Difficulty**: Tool-augmented visual reasoning ### Design2Code (Design2Code) A multimodal coding benchmark for turning visual designs into working frontend implementations. - **Year**: 2026 - **Tasks**: Design-to-code tasks - **Format**: Visual input to frontend implementation - **Difficulty**: Multimodal coding ### Flame-VLM-Code (Flame-VLM-Code) A vision-language coding benchmark for generating correct code from visual and multimodal inputs. - **Year**: 2026 - **Tasks**: Multimodal coding tasks - **Format**: Vision-language code generation - **Difficulty**: Multimodal coding ### Vision2Web (Vision2Web) A benchmark for converting visual references into functional web implementations. - **Year**: 2026 - **Tasks**: Screenshot-to-web tasks - **Format**: Visual reference to web implementation - **Difficulty**: Multimodal web generation ### ImageMining (ImageMining) A multimodal retrieval and extraction benchmark over image-heavy task settings. - **Year**: 2026 - **Tasks**: Visual retrieval tasks - **Format**: Image-grounded retrieval and extraction - **Difficulty**: Multimodal retrieval ### MMSearch (MMSearch) A multimodal search benchmark for retrieval and grounded answering across mixed-media inputs. - **Year**: 2026 - **Tasks**: Multimodal search tasks - **Format**: Mixed-media retrieval and grounded answering - **Difficulty**: Multimodal search ### MMSearch-Plus (MMSearch-Plus) A harder MMSearch variant for multimodal retrieval and grounded tool-use workflows. - **Year**: 2026 - **Tasks**: Hard multimodal search tasks - **Format**: Advanced mixed-media retrieval benchmark - **Difficulty**: Advanced multimodal search ### SimpleVQA (SimpleVQA) A visual question answering benchmark focused on straightforward image-grounded understanding. - **Year**: 2026 - **Tasks**: Visual QA tasks - **Format**: Image-grounded question answering - **Difficulty**: General visual understanding ### Facts-VLM (Facts-VLM) A grounded multimodal factuality benchmark for evidence-linked answer correctness. - **Year**: 2026 - **Tasks**: Grounded factuality tasks - **Format**: Evidence-linked multimodal factuality - **Difficulty**: Grounded multimodal factuality ### V* (V*) A vision-centric benchmark for high-level multimodal reasoning and perception quality. - **Year**: 2026 - **Tasks**: Frontier multimodal reasoning tasks - **Format**: Vision-centric reasoning benchmark - **Difficulty**: Frontier multimodal ### CharXiv (CharXiv Reasoning) A scientific chart reasoning benchmark that tests whether models can understand, interpret, and reason about complex scientific visualizations including plots, diagrams, and data charts. - **Year**: 2024 - **Tasks**: Scientific chart reasoning - **Format**: Chart understanding and reasoning - **Difficulty**: Scientific visualization reasoning ### CharXiv w/o tools (CharXiv Reasoning without tools) Tool-free variant of CharXiv that isolates raw visual reasoning ability without code execution or tool augmentation. - **Year**: 2024 - **Tasks**: Scientific chart reasoning (tool-free) - **Format**: Chart understanding without tools - **Difficulty**: Scientific visualization reasoning ### BabyVision (BabyVision) A multimodal benchmark for fine-grained visual perception and grounded reasoning tasks. - **Year**: 2026 - **Tasks**: Visual perception tasks - **Format**: Multimodal visual reasoning - **Difficulty**: Fine-grained visual perception ### SWE-bench Multimodal (SWE-bench Multimodal) A multimodal variant of SWE-bench that adds visual context (screenshots, design mockups) to software engineering issue descriptions, testing whether models can leverage visual information for code generation. - **Year**: 2025 - **Tasks**: Multimodal software engineering tasks - **Format**: Code patch generation with visual context - **Difficulty**: Frontier multimodal coding ### Blueprint-Bench 2 (Blueprint-Bench 2) An agentic spatial reasoning benchmark reported as a normalized score. - **Year**: 2026 - **Tasks**: Spatial reasoning from blueprints - **Format**: Normalized score - **Difficulty**: Agentic spatial reasoning ### CharXiv (overall) (CharXiv Descriptive and Reasoning Combined) A scientific chart understanding benchmark scored across both the descriptive and reasoning splits rather than the reasoning split alone. - **Year**: 2024 - **Tasks**: Scientific chart description and reasoning - **Format**: Chart understanding and reasoning - **Difficulty**: Scientific visualization reasoning ### A-OKVQA (A Benchmark for Visual Question Answering using World Knowledge) A visual question answering benchmark whose questions cannot be answered from the image alone and require commonsense or world knowledge. - **Year**: 2022 - **Tasks**: Knowledge-grounded visual question answering - **Format**: Multiple choice and direct answer - **Difficulty**: Commonsense and world knowledge about images ### OmniDocBench 1.6 (OmniDocBench v1.6) A document parsing benchmark whose overall score composites edit distance, CDM and TEDS sub-scores across diverse PDF layouts. - **Year**: 2024 - **Tasks**: Document parsing and extraction - **Format**: Composite of edit distance, CDM and TEDS - **Difficulty**: Grounded document reasoning ### AudioMC (Scale Labs AudioMC) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 35 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### AudioMC – Audio Output (Scale Labs AudioMC – Audio Output) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 15 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### AudioMC – Text Output (Scale Labs AudioMC – Text Output) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 18 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### Humanity's Sixth Sense (HSS) (Humanity's Sixth Sense: Benchmarking Intuitive Visual Reasoning in Multimodal Models) HSS tests what a model can infer from images and silent videos across temporal, spatial, social, and abstract reasoning. We show the published results as display-only evidence, excluded from overall and category rankings. - **Year**: 2026 - **Tasks**: 522 tasks (288 image, 234 video) - **Format**: Pass@1 (%) averaged over three attempts per task - **Difficulty**: Intuitive visual reasoning ### Visual-Language Understanding (Scale Labs Visual-Language Understanding) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 63 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### ViDoRe V3 Image (Perplexity report) (ViDoRe V3 Image (Perplexity report)) Perplexity reports public ViDoRe V3 image retrieval nDCG@10 of 62.3 for the 0.6B model and 65.2 for the 9B model. - **Year**: 2026 - **Format**: Public ViDoRe V3 retrieval - **Difficulty**: Visual document retrieval ### ViDoRe V3 Markdown (Perplexity report) (ViDoRe V3 Markdown (Perplexity report)) Perplexity reports public ViDoRe V3 Markdown retrieval nDCG@10 of 61.2 for the 0.6B model and 64.7 for the 9B model. - **Year**: 2026 - **Format**: Public ViDoRe V3 retrieval - **Difficulty**: Visual document retrieval ### Vals Multimodal Index (Vals Multimodal Index v1.2) Vals AI multimodal composite across finance, coding, education, and mortgage-tax task families. - **Year**: 2026 - **Tasks**: Finance, coding, education, and mortgage-tax components - **Format**: Composite score - **Difficulty**: Private multimodal economic-work benchmark composite ### MortgageTax (Vals MortgageTax) Vals AI benchmark for mortgage and tax document reasoning, including semantic and numerical extraction task views. - **Year**: 2026 - **Tasks**: Mortgage and tax extraction tasks - **Format**: Accuracy score - **Difficulty**: Professional mortgage-tax document reasoning ### VoxelBench Text (VoxelBench Text-Prompt Leaderboard) A live human-preference benchmark where language models turn text prompts into voxel structures and voters compare anonymous builds from the same prompt. - **Year**: 2025 - **Tasks**: Live text prompts for 3D voxel construction - **Format**: Glicko-2 rating from blind pairwise votes - **Difficulty**: 3D spatial construction and visual quality ### VoxelBench Image (VoxelBench Image-Prompt Leaderboard) A live human-preference benchmark where multimodal models build voxel structures from image references and voters compare anonymous results produced from the same prompt. - **Year**: 2025 - **Tasks**: Live image-reference prompts for 3D voxel construction - **Format**: Glicko-2 rating from blind pairwise votes - **Difficulty**: Visual grounding, 3D construction, and aesthetic quality ### VoiceCodeBench (Vals VoiceCodeBench) Tests whether speech-to-text models preserve exact structured values in English workplace audio. - **Year**: 2026 - **Tasks**: 300 workplace speech clips with exact structured values - **Format**: Task success rate - **Difficulty**: Structured-value speech transcription ### SAGE (Vals SAGE) Student Assessment with Generative Evaluation - **Year**: 2026 - **Tasks**: Student assessment with generative evaluation - **Format**: Accuracy score - **Difficulty**: Education assessment reasoning ### MathVista (MathVista) MathVista tests mathematical reasoning about images. This table shows published accuracy on the testmini split, with each source configuration kept separate. The results are display only and do not enter overall or category rankings. - **Year**: 2023 - **Tasks**: 1,000 examples in the testmini split - **Format**: Published accuracy (%) - **Difficulty**: Mathematical reasoning in visual contexts ### MathVista test (MathVista test) MathVista tests mathematical reasoning about images. This table shows published accuracy on the test split, with each source configuration kept separate. The results are display only and do not enter overall or category rankings. - **Year**: 2023 - **Tasks**: 5,141 examples in the test split - **Format**: Published accuracy (%) - **Difficulty**: Mathematical reasoning in visual contexts ### DocVQA (DocVQA) DocVQA asks questions about a single document image. This page shows four historical Anthropic reports on the test split, with ANLS, zero-shot setup, report dates, and source cohorts kept visible. - **Year**: 2021 - **Tasks**: Questions about single document images; provider run counts undisclosed - **Format**: Text answers drawn from document images ### ChartQA (ChartQA) ChartQA tests visual and logical reasoning about charts. These own-provider reports keep Anthropic's relaxed-accuracy results separate from Zyphra scores whose metric and scorer are not disclosed. - **Year**: 2022 - **Format**: Provider-reported aggregates; separate source cohorts ### InfoVQA (InfographicVQA) InfographicVQA asks questions about infographic images. These two own-provider InfoVQA test reports name the split but do not disclose the metric or scorer; their printed scores remain separate source facts. - **Year**: 2021 - **Format**: Provider-reported scores; metric/scorer not disclosed ### TextVQA (TextVQA) TextVQA requires reading and reasoning about text in images. These two own-provider reports use the validation split, with metric and scorer left undisclosed in the original model cards. - **Year**: 2019 - **Format**: Provider-reported validation scores; metric/scorer not disclosed ### CharXiv Extended (Mercor) (CharXiv Extended — Mercor evaluation) Answering descriptive and reasoning questions about research-paper charts. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 150 Mercor tasks; 5000 tasks in the separately reported original set - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### MedXpertQA MM Extended (Mercor) (MedXpertQA MM Extended — Mercor evaluation) Medical multiple-choice reasoning over images and clinical details. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 51 Mercor tasks; 200 tasks in the separately reported original set - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### MMMU-Pro Extended (Mercor) (MMMU-Pro Extended — Mercor evaluation) Answering questions that require both visual and textual understanding. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 140 Mercor tasks; 1730 tasks in the separately reported original set - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### CharXiv (Mercor) (CharXiv — Mercor evaluation) Answering descriptive and reasoning questions about research-paper charts. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 5,000 source-reported tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### MedXpertQA MM (Mercor) (MedXpertQA MM — Mercor evaluation) Medical multiple-choice reasoning over images and clinical details. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 200 source-reported tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ## Knowledge Benchmarks ### HealthBench (raw) (HealthBench raw score) Raw score on realistic multi-turn healthcare conversations graded against expert-written rubrics. - **Year**: 2026 - **Tasks**: 5,000 multi-turn patient conversations - **Format**: Raw rubric score - **Difficulty**: Realistic healthcare conversations ### HealthBench (length-adjusted) (HealthBench length-adjusted score) HealthBench score after applying a verbosity penalty to model responses. - **Year**: 2026 - **Tasks**: 5,000 multi-turn patient conversations - **Format**: Length-adjusted rubric score - **Difficulty**: Realistic healthcare conversations ### HealthBench Professional (raw) (HealthBench Professional raw score) Raw score on physician-authored clinical consult, documentation, and research conversations. - **Year**: 2026 - **Tasks**: 525 physician-authored conversations - **Format**: Raw rubric score - **Difficulty**: Professional clinical tasks ### BioMysteryBench (human-solvable) (BioMysteryBench Human Solvable) Computational biology challenges that independent human experts were able to solve. - **Year**: 2026 - **Tasks**: Human-solvable computational biology investigations - **Format**: Task score - **Difficulty**: Expert computational biology ### BioMysteryBench (human-difficult) (BioMysteryBench Human Difficult) Computational biology challenges with objective answers that remained unsolved by independent human experts. - **Year**: 2026 - **Tasks**: Human-difficult computational biology investigations - **Format**: Task score - **Difficulty**: Frontier computational biology ### SpatialBench Verified (LatchBio SpatialBench Verified) Analysis of spatial transcriptomics data across externally validated biological problems. - **Year**: 2026 - **Tasks**: 115 externally validated spatial transcriptomics problems - **Format**: Task score - **Difficulty**: Professional bioinformatics ### SingleCellBench (LatchBio SingleCellBench) Single-cell RNA sequencing analysis tasks spanning common bioinformatics workflows. - **Year**: 2026 - **Tasks**: 195 single-cell RNA sequencing problems - **Format**: Task score - **Difficulty**: Professional bioinformatics ### ProteinGym Hard (ProteinGym Hard) Predicts mutation effects by ranking mutant protein sequences against wild type and comparing against laboratory measurements. - **Year**: 2026 - **Tasks**: Hard protein mutation-effect ranking tasks - **Format**: Rank correlation - **Difficulty**: Computational protein science ### Protein Design (Anthropic Protein Design evaluation) Generates novel protein sequences under family, topology, globularity, and structural-motif constraints. - **Year**: 2026 - **Tasks**: Constrained protein-sequence design tasks - **Format**: Composite score - **Difficulty**: Computational protein design ### Morphology-to-molecule matching (Axiom Bio morphology-to-molecule matching) Matches multi-channel cellular imaging data to the molecule-dose perturbations that produced it. - **Year**: 2026 - **Tasks**: Four multi-channel cell-imaging perturbation-matching tasks - **Format**: Matching accuracy - **Difficulty**: Biomedical image and molecular reasoning ### Medicinal chemistry (Anthropic medicinal-chemistry evaluation) Selects compounds with the highest or lowest measured ADME-related property from molecular structures and reference values. - **Year**: 2026 - **Tasks**: 504 molecular-property questions - **Format**: Accuracy - **Difficulty**: Medicinal chemistry reasoning ### Protein Design library ranking (Anthropic protein-design library-ranking task) Ranks held-out protein designs for experimental testing using wet-lab campaign measurements and a selection objective. - **Year**: 2026 - **Tasks**: 83 protein-design library-ranking problems - **Format**: Chance-normalized average precision - **Difficulty**: Computational protein design ### De novo protein-binder design (Anthropic de novo protein-binder design evaluation) Designs diverse miniprotein binders against target protein structures in a GPU-equipped research sandbox. - **Year**: 2026 - **Tasks**: 15 protein targets - **Format**: Composite structural design score - **Difficulty**: Long-horizon computational protein design ### Organic chemistry V2 (Anthropic Organic Chemistry V2 evaluation) Chemistry tasks covering spectroscopy, synthesis planning, reaction prediction, and chemical structure images. - **Year**: 2026 - **Tasks**: Organic chemistry reasoning tasks - **Format**: Task score - **Difficulty**: Expert organic chemistry ### Protocols (troubleshooting) (Molecular Biology Protocols Troubleshooting) Detects and fixes errors in molecular-biology protocols using document, code, and web-search tools. - **Year**: 2026 - **Tasks**: Molecular-biology protocol troubleshooting - **Format**: Task score - **Difficulty**: Expert laboratory protocols ### Protocols (understanding) (Benchling Molecular Biology Protocols Understanding) Extends online molecular-biology protocols in additional directions using document and web-search tools. - **Year**: 2026 - **Tasks**: Molecular-biology protocol extension - **Format**: Task score - **Difficulty**: Expert laboratory protocols ### AA Openness Index (Artificial Analysis Openness Index) A display-only Artificial Analysis model-openness index. - **Year**: 2026 - **Tasks**: Model openness assessment - **Format**: Index score - **Difficulty**: Display-only external reference ### MMLU (Massive Multitask Language Understanding) A comprehensive multiple-choice question answering test covering 57 tasks including elementary mathematics, US history, computer science, law, and more. Tests knowledge across diverse academic subjects from high school to professional level. - **Year**: 2020 - **Tasks**: 57 subjects - **Format**: Multiple choice questions - **Difficulty**: Elementary to professional level ### GPQA (Graduate-Level Google-Proof Q&A) A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Designed to be difficult even for skilled non-experts with access to Google. - **Year**: 2023 - **Tasks**: 448 questions - **Format**: Multiple choice questions - **Difficulty**: Graduate level ### GPQA-D (GPQA Diamond) A display-only GPQA Diamond reference from provider comparison charts. - **Year**: 2023 - **Tasks**: Graduate-level science questions - **Format**: Multiple choice questions - **Difficulty**: Graduate level ### SuperGPQA (SuperGPQA: Scaling LLM Evaluation Across 285 Graduate Disciplines) An expanded version of GPQA that evaluates graduate-level knowledge and reasoning capabilities across 285 disciplines, providing comprehensive coverage of academic domains. - **Year**: 2025 - **Tasks**: 285 disciplines - **Format**: Multiple choice questions - **Difficulty**: Graduate level ### MMLU-Pro (Massive Multitask Language Understanding Professional) An enhanced version of MMLU with 10 answer choices instead of 4, featuring more reasoning-focused questions that better differentiate frontier models. - **Year**: 2024 - **Tasks**: Multiple subjects - **Format**: 10-way multiple choice - **Difficulty**: Professional level ### AGIEval (AGIEval) A human-centric exam benchmark for general knowledge and reasoning reported in DeepSeek-V4 base-model evaluations. - **Year**: 2026 - **Tasks**: General academic and professional exam questions - **Format**: Exact match - **Difficulty**: General knowledge ### HLE (Humanity's Last Exam) An expert-authored benchmark designed to probe frontier knowledge and reasoning. BenchLM keeps protocol differences visible because tool-assisted and closed-book HLE runs answer different questions. - **Year**: 2025 - **Tasks**: Expert-level questions - **Format**: Open-ended and multiple choice - **Difficulty**: Frontier expert level ### HLE-Diamond (Humanity's Last Exam Diamond) Accuracy on the 1,000-question HLE-Diamond set without tools, using the benchmark owner's high-reasoning evaluation. - **Year**: 2026 - **Tasks**: 1,000 expert questions: 500 reasoning and 500 knowledge - **Format**: Accuracy without tools, reasoning high - **Difficulty**: Frontier expert knowledge and reasoning ### HLE-Diamond (highest effort) (Humanity's Last Exam Diamond without tools at highest reasoning effort) Accuracy on the 1,000-question HLE-Diamond set without tools at each model's highest available reasoning effort. - **Year**: 2026 - **Tasks**: 1,000 expert questions: 500 reasoning and 500 knowledge - **Format**: Accuracy without tools, highest available reasoning effort - **Difficulty**: Frontier expert knowledge and reasoning ### HLE-Diamond (web+code) (Humanity's Last Exam Diamond with web and code tools) Accuracy on HLE-Diamond with constrained web search, page fetching, and code execution at high reasoning effort. - **Year**: 2026 - **Tasks**: 1,000 expert questions with web and code tools - **Format**: Accuracy with web and code, reasoning high - **Difficulty**: Frontier expert knowledge and reasoning ### HLE-Verified (HLE-Verified) A verified and revised Humanity's Last Exam set that removes uncertain items and repairs fixable questions before evaluating frontier expert reasoning. - **Year**: 2026 - **Tasks**: 1,811 verified or revised expert questions - **Format**: Full-set accuracy - **Difficulty**: Frontier multidisciplinary expert reasoning ### LABBench2 (LABBench2: An Improved Benchmark for AI Systems Performing Biology Research) A benchmark of realistic biology-research tasks involving literature, figures, tables, databases, and bioinformatics files. - **Year**: 2026 - **Tasks**: Nearly 1,900 biology-research tasks - **Format**: Aggregate accuracy - **Difficulty**: Real-world biology research ### FrontierScience (FrontierScience) A benchmark for research-level scientific reasoning, designed to separate frontier models on difficult science tasks that mix domain knowledge with deep reasoning. - **Year**: 2026 - **Tasks**: Research-level science tasks - **Format**: Scientific reasoning benchmark - **Difficulty**: Research frontier ### Artificial Analysis Intelligence Index (Artificial Analysis Intelligence Index) A display-only intelligence index published by Artificial Analysis that aggregates provider-reported and benchmark-derived signals into a single model-level score. - **Year**: 2026 - **Tasks**: Cross-benchmark intelligence index - **Format**: Aggregated model score - **Difficulty**: Display-only external reference ### AA-HLE (Artificial Analysis Humanity's Last Exam) An Artificial Analysis Humanity's Last Exam score. - **Year**: 2026 - **Tasks**: Expert-level questions - **Format**: Accuracy - **Difficulty**: Frontier expert reasoning ### AA-Omniscience Index (Artificial Analysis Omniscience Index) A display-only Artificial Analysis factual knowledge index. - **Year**: 2026 - **Tasks**: Knowledge questions - **Format**: Index score - **Difficulty**: Broad factual knowledge ### AA-Omniscience Accuracy (Artificial Analysis Omniscience Accuracy) An Artificial Analysis knowledge metric for the proportion of correctly answered questions. - **Year**: 2026 - **Tasks**: Knowledge questions - **Format**: Accuracy - **Difficulty**: Broad knowledge ### AA-Omniscience Hallucination Rate (Artificial Analysis Omniscience Hallucination Rate) A display-only Artificial Analysis factuality metric for the rate of incorrect answers among non-correct responses. - **Year**: 2026 - **Tasks**: Knowledge questions - **Format**: Hallucination rate - **Difficulty**: Factuality ### SimpleQA (Measuring Short-Form Factuality in Large Language Models) A benchmark that evaluates the ability of language models to answer short, fact-seeking questions accurately. Focuses on factual correctness rather than reasoning complexity. - **Year**: 2024 - **Tasks**: Factual questions - **Format**: Short-form Q&A - **Difficulty**: Factual accuracy focused ### Chinese-SimpleQA (Chinese-SimpleQA) A Chinese short-form factuality benchmark reported by DeepSeek for V4 model evaluations. - **Year**: 2026 - **Tasks**: Chinese factual questions - **Format**: Short-form factual QA - **Difficulty**: Factual accuracy focused ### OpenBookQA (OpenBookQA) A science question-answering benchmark that tests whether models can apply a small open-book set of elementary science facts to multi-step reasoning questions. - **Year**: 2018 - **Tasks**: Elementary science questions - **Format**: 4-way multiple choice - **Difficulty**: Elementary science reasoning ### HealthBench Hard (HealthBench Hard) A harder subset of OpenAI's HealthBench for evaluating open-ended medical and health reasoning with rubric-based grading. - **Year**: 2026 - **Tasks**: 1,000 health prompts - **Format**: Open-ended health evaluation - **Difficulty**: Advanced health reasoning ### HealthBench Professional (HealthBench Professional) An open benchmark for clinician-facing model responses across care consult, writing and documentation, and medical research tasks. - **Year**: 2026 - **Tasks**: Clinician chat tasks - **Format**: Rubric-graded open-ended responses - **Difficulty**: Professional clinical workflows ### MedXpertQA (Text) (MedXpertQA Text) A medical multiple-choice benchmark spanning many specialties with 10 answer options per question. - **Year**: 2026 - **Tasks**: 2,450 medical multiple-choice questions - **Format**: Medical MCQ - **Difficulty**: Professional medical knowledge ### FrontierScience Research (FrontierScience Research) A research-focused FrontierScience evaluation variant for scientific investigation and problem solving. - **Year**: 2026 - **Tasks**: Scientific research problems - **Format**: Research evaluation - **Difficulty**: Frontier scientific research ### TruthfulQA (TruthfulQA) A benchmark designed to measure whether language models produce truthful answers instead of repeating common misconceptions or misleading falsehoods. - **Year**: 2021 - **Tasks**: Truthfulness and misconception resistance - **Format**: Question answering - **Difficulty**: Hallucination and factuality stress test ### HLE w/o tools (Humanity's Last Exam without tools) Tool-free variant of Humanity's Last Exam that isolates a model's raw frontier reasoning. - **Year**: 2026 - **Tasks**: Expert-level questions - **Format**: Tool-free expert QA - **Difficulty**: Frontier expert level ### MMLU-Redux (MMLU-Redux) A harder refresh of MMLU intended to keep broad knowledge evaluation useful after the original benchmark became too easy for frontier models. - **Year**: 2026 - **Tasks**: Broad academic QA - **Format**: Multiple choice questions - **Difficulty**: Advanced general knowledge ### MMMLU (MMMLU) A multilingual MMLU-style benchmark reported in provider evaluation tables. - **Year**: 2026 - **Tasks**: Multilingual academic QA - **Format**: Exact match - **Difficulty**: Broad multilingual knowledge ### C-Eval (C-Eval) A Chinese-language academic and professional benchmark spanning humanities, social science, STEM, and applied subjects. - **Year**: 2023 - **Tasks**: Chinese academic and professional exams - **Format**: Multiple choice questions - **Difficulty**: High school to professional level ### CMMLU (Chinese Massive Multitask Language Understanding) A Chinese multitask academic benchmark reported in DeepSeek-V4 base-model evaluations. - **Year**: 2026 - **Tasks**: Chinese academic QA - **Format**: Exact match - **Difficulty**: Broad Chinese knowledge ### MultiLoKo (MultiLoKo) A multilingual/localized knowledge benchmark reported in DeepSeek-V4 base-model evaluations. - **Year**: 2026 - **Tasks**: Localized multilingual knowledge questions - **Format**: Exact match - **Difficulty**: Multilingual knowledge ### FACTS Parametric (FACTS Parametric) A parametric factuality benchmark reported in DeepSeek-V4 base-model evaluations. - **Year**: 2026 - **Tasks**: Parametric factual recall - **Format**: Exact match - **Difficulty**: Factual accuracy focused ### TriviaQA (TriviaQA) A reading and trivia question-answering benchmark reported in DeepSeek-V4 base-model evaluations. - **Year**: 2026 - **Tasks**: Trivia and reading-comprehension QA - **Format**: Exact match - **Difficulty**: General factual QA ### FinanceArena (FinanceArena — FinanceQA Assumption-Based) An AfterQuery benchmark of open-ended financial analysis that requires models to read financial data, make assumptions, and return exact answers. - **Year**: 2025 - **Tasks**: Professional financial-analysis questions - **Format**: Open-ended financial QA with exact-match grading - **Difficulty**: Professional finance reasoning ### DrugDiscoveryBench (Scale Labs DrugDiscoveryBench) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 23 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### FORTRESS (Scale Labs FORTRESS) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 70 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### Humanity's Last Exam (Diamond) (Scale Labs Humanity's Last Exam (Diamond)) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 13 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### Humanity's Last Exam (Scale Labs Humanity's Last Exam) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 55 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### Humanity's Last Exam (Text Only) (Scale Labs Humanity's Last Exam (Text Only)) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 65 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### MultiNRC (Scale Labs MultiNRC) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 44 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### PRBench Finance (Scale Labs PRBench Finance) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 37 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### PRBench Legal (Scale Labs PRBench Legal) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 41 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### SciPredict (Scale Labs SciPredict) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 15 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### TutorBench (Scale Labs TutorBench) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 27 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### CorpFin v2 (Vals CorpFin v2) Vals AI private benchmark for understanding long-context credit agreements. - **Year**: 2026 - **Tasks**: Credit-agreement understanding tasks - **Format**: Accuracy score - **Difficulty**: Professional finance document reasoning ### MedCode (Vals MedCode) Vals AI healthcare benchmark for whether models can support the medical billing process. - **Year**: 2026 - **Tasks**: Medical billing support tasks - **Format**: Accuracy score - **Difficulty**: Professional healthcare administration ### MedScribe (Vals MedScribe) Vals AI healthcare benchmark for whether models can support doctors with administrative work. - **Year**: 2026 - **Tasks**: Medical administrative support tasks - **Format**: Accuracy score - **Difficulty**: Professional healthcare administration ### LegalBench (Vals LegalBench) Vals AI legal benchmark with issue, rule, conclusion, interpretation, and rhetoric task views. - **Year**: 2026 - **Tasks**: Legal reasoning task views - **Format**: Accuracy score - **Difficulty**: Professional legal reasoning ### CaseLaw v2 (Vals CaseLaw v2) Vals AI private question-answer benchmark over Canadian court cases. - **Year**: 2026 - **Tasks**: Canadian case-law question answering - **Format**: Accuracy score - **Difficulty**: Professional legal retrieval and reasoning ### TaxEval v2 (Vals TaxEval v2) A Vals-created set of questions and responses to tax questions - **Year**: 2026 - **Tasks**: Tax question answering and response evaluation - **Format**: Accuracy score - **Difficulty**: Professional tax reasoning ### MedQA (Vals MedQA) Evaluating language model bias in medical questions. - **Year**: 2026 - **Tasks**: Medical question answering - **Format**: Accuracy score - **Difficulty**: Medical knowledge and bias evaluation ### SimpleQA Verified (SimpleQA Verified) SimpleQA Verified measures short-form factual recall on 1,000 prompts without tools. This dated author-paper table ranks source models by F1 and keeps accuracy, accuracy given attempted, attempted, and hedged separate. - **Year**: 2025 - **Tasks**: 1,000 prompts - **Format**: Short-form factual answers ### HLE Extended (Mercor) (HLE Extended — Mercor evaluation) Answering difficult academic questions across many subjects. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 447 Mercor tasks; 2158 tasks in the separately reported original set - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### HLE (Mercor) (HLE — Mercor evaluation) Answering difficult academic questions across many subjects. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 2,158 source-reported tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ### SUPERChem (Mercor) (SUPERChem — Mercor evaluation) Answering advanced chemistry questions. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 500 source-reported tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ## Multilingual Benchmarks ### GMMLU (Global MMLU) MMLU-style knowledge evaluation across 42 high- and low-resource languages. - **Year**: 2024 - **Tasks**: Knowledge questions across 42 languages - **Format**: Average accuracy - **Difficulty**: Multilingual knowledge ### MILU (Multi-task Indic Language Understanding Benchmark) Culturally grounded knowledge comprehension across ten Indic languages and English. - **Year**: 2024 - **Tasks**: Knowledge tasks across 11 languages - **Format**: Average accuracy - **Difficulty**: Multilingual Indic knowledge ### AA Global-MMLU-Lite (Artificial Analysis Global-MMLU-Lite) An independently evaluated multilingual knowledge result from Artificial Analysis. - **Year**: 2026 - **Tasks**: Multilingual knowledge questions - **Format**: Accuracy - **Difficulty**: Multilingual professional knowledge ### MGSM (Multilingual Grade School Math) A multilingual benchmark that translates 250 grade school math problems from GSM8K into 10 typologically diverse languages: Bengali, German, Spanish, French, Japanese, Russian, Swahili, Telugu, Thai, and Chinese. - **Year**: 2022 - **Tasks**: 250 problems × 11 languages - **Format**: Math word problems - **Difficulty**: Grade school math, multilingual ### MMLU-ProX (MMLU-ProX) A multilingual extension of professional-level academic evaluation across many languages. - **Year**: 2025 - **Tasks**: Multilingual professional QA - **Format**: Multilingual multiple choice - **Difficulty**: Professional multilingual ### NOVA-63 (NOVA-63) A broad multilingual benchmark row from Qwen's launch comparisons intended to measure cross-lingual capability beyond a single language family. - **Year**: 2026 - **Tasks**: Broad multilingual evaluation - **Format**: Cross-lingual benchmark - **Difficulty**: Broad multilingual capability ### INCLUDE (INCLUDE) A multilingual benchmark used in provider tables to measure inclusive language coverage and cross-lingual understanding beyond common high-resource languages. - **Year**: 2026 - **Tasks**: Cross-lingual understanding - **Format**: Multilingual benchmark - **Difficulty**: Broad multilingual capability ### PolyMath (PolyMath) A multilingual mathematical reasoning benchmark that tests whether math performance transfers across languages rather than only in English. - **Year**: 2026 - **Tasks**: Multilingual math problems - **Format**: Cross-lingual mathematical reasoning - **Difficulty**: Advanced multilingual reasoning ### VWT2k-lite (VWT2k-lite) A lighter multilingual benchmark slice published in provider tables for broad cross-lingual transfer and understanding. - **Year**: 2026 - **Tasks**: Multilingual transfer tasks - **Format**: Cross-lingual benchmark - **Difficulty**: Broad multilingual capability ### MAXIFE (MAXIFE) A multilingual instruction-following and understanding benchmark row published in Qwen's launch comparisons. - **Year**: 2026 - **Tasks**: Multilingual instruction following - **Format**: Cross-lingual benchmark - **Difficulty**: Advanced multilingual instruction following ### SWE Multilingual (SWE-bench Multilingual) A multilingual extension of SWE-bench covering 300 problems across 9 programming languages, testing code generation and bug fixing beyond Python. - **Year**: 2025 - **Tasks**: 300 problems across 9 languages - **Format**: Multi-language code patch generation - **Difficulty**: Professional multilingual software engineering ### NanoBEIR Multilingual (NanoBEIR Multilingual Extended) A display-only multilingual retrieval benchmark reported by Liquid AI for LFM2.5 retriever models, using NDCG@10 across 11 languages. - **Year**: 2026 - **Tasks**: Multilingual document retrieval - **Format**: NDCG@10 average - **Difficulty**: Multilingual retrieval ### MKQA-11 (MKQA-11 multilingual retrieval) A display-only multilingual QA retrieval benchmark reported by Liquid AI for LFM2.5 retriever models, using Recall@20 across 11 languages. - **Year**: 2026 - **Tasks**: Cross-lingual open-domain QA retrieval - **Format**: Recall@20 average - **Difficulty**: Multilingual retrieval ### WMT26 (WMT26 General Machine Translation (all languages)) Machine-translation quality on the WMT26 general translation task, averaged across all evaluated languages and scored 0-100 by an LLM judge. - **Year**: 2026 - **Tasks**: General-domain translation across 50+ languages (32 high-resource plus 18 additional) - **Format**: 0-100 judged translation quality, averaged across languages - **Difficulty**: Professional machine translation ### Long-context translation (Long-context translation evaluation (Cohere)) Cohere's long-document translation evaluation: two book chapters translated in a single call, scored per paragraph with xCOMET-XL. - **Year**: 2026 - **Tasks**: Two book chapters per call - **Format**: xCOMET-XL quality score - **Difficulty**: Long-document translation ## Instruction Following Benchmarks ### Gray Swan IPI (15 attempts) (Gray Swan Indirect Prompt Injection, attack success at 15 attempts) Measures how often indirect prompt-injection attacks succeed within fifteen attempts. Lower scores mean greater resistance to the tested attacks. - **Year**: 2026 - **Tasks**: Indirect prompt-injection attacks - **Format**: Attack success rate at 15 attempts - **Difficulty**: Adversarial prompt injection ### IFEval (Instruction-Following Eval) A benchmark of 541 prompts built from 25 verifiable instruction types. It tests whether a model follows checkable constraints such as keyword, length, casing, and response-format requirements. - **Year**: 2023 - **Tasks**: 541 prompts across 25 instruction types - **Format**: Constrained generation - **Difficulty**: Instruction precision ### IFBench (Instruction Following Benchmark) IFBench evaluates precise instruction-following generalization on 58 challenging, verifiable out-of-domain constraints. Unlike IFEval which tests familiar constraint types, IFBench specifically measures how well models follow novel instructions they haven't been optimized for, exposing overfitting to common instruction patterns. - **Year**: 2025 - **Tasks**: 58 ### SOB Value Acc (Structured Output Benchmark Value Accuracy) A structured-output benchmark from Interfaze measuring whether extracted JSON leaf values exactly match verified ground truth. - **Year**: 2026 - **Tasks**: Structured output extraction - **Format**: Value accuracy - **Difficulty**: Production structured-output reliability ### MASK (Scale Labs MASK) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 67 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### MultiChallenge (Scale Labs MultiChallenge) A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings. - **Year**: 2026 - **Tasks**: 32 published rows - **Format**: Published Scale leaderboard score - **Difficulty**: External agent and model evaluation ### KindBench (KindBench Psychological Safety Benchmark) A behavioral benchmark that tests psychological safety across sixteen adversarial multi-turn conversations covering emotional safety, identity, sycophancy, and value integrity. - **Year**: 2026 - **Tasks**: 16 multi-turn scenarios, 72 criteria - **Format**: Judge-scored behavioral audit with human review - **Difficulty**: Adversarial psychological-safety evaluation ### ACCR standard (Advanced Cyber Completion Rate — Standard Access) Share of approved advanced-cyber requests completed rather than refused under standard GPT-5.6 Sol safeguards. - **Year**: 2026 - **Tasks**: Internal advanced-cyber request set - **Format**: Completion rate - **Difficulty**: Cyber access and safeguards ### ACCR Daybreak Blue (Advanced Cyber Completion Rate — Daybreak Blue) Share of approved advanced-cyber requests completed rather than refused by GPT-5.6 Sol under Daybreak Blue safeguards. - **Year**: 2026 - **Tasks**: Internal advanced-cyber request set - **Format**: Completion rate - **Difficulty**: Cyber access and safeguards ### ACCR Daybreak Red (Advanced Cyber Completion Rate — Daybreak Red) Share of approved advanced-cyber requests completed rather than refused by GPT-5.6 Cyber under Daybreak Red access. - **Year**: 2026 - **Tasks**: Internal advanced-cyber request set - **Format**: Completion rate - **Difficulty**: Cyber access and safeguards ### WritingBench (WritingBench) WritingBench evaluates generative writing across six domains. This dated benchmark-owner table contains 54 source model rows, with overall and domain scores on the native 0-100 scale. - **Year**: 2025 - **Tasks**: 1,000 queries - **Format**: Judge score, 0-100 points ### Multi-IF (Multi-IF) Multi-IF tests instruction following over three turns in eight languages. This selected historical author report preserves each turn score, dated API identifier, and generation settings for four models. - **Year**: 2024 - **Tasks**: 4,501 three-turn conversations in eight languages - **Format**: Turn-specific accuracy average on a 0-1 scale ### AdvancedIF (Mercor) (AdvancedIF — Mercor evaluation) Following complex instructions graded against rubric criteria. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 1,645 source-reported tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ## Mathematics Benchmarks ### IMO 2026 (International Mathematical Olympiad 2026) Proof-based olympiad performance on all six IMO 2026 problems. - **Year**: 2026 - **Tasks**: 6 proof-based problems - **Format**: Official-style proof score - **Difficulty**: International olympiad mathematics ### RiemannBench (no tools) (RiemannBench without tools) Research-level mathematics problems with unique programmatically verified closed-form answers. - **Year**: 2026 - **Tasks**: 25 private research-level mathematics problems - **Format**: Accuracy without tools - **Difficulty**: Research mathematics ### RiemannBench (tools) (RiemannBench with tools) Research-level mathematics problems with unique programmatically verified closed-form answers and tool access. - **Year**: 2026 - **Tasks**: 25 private research-level mathematics problems - **Format**: Accuracy with tools - **Difficulty**: Research mathematics ### ArXivMath Jun. 2026 (no tools) (ArXivMath June 2026 without tools) Final-answer research mathematics problems drawn from recent arXiv abstracts. - **Year**: 2026 - **Tasks**: 49 recent research-mathematics problems - **Format**: Final-answer accuracy - **Difficulty**: Research mathematics ### ArXivMath Jun. 2026 (tools) (ArXivMath June 2026 with tools) Final-answer research mathematics problems drawn from recent arXiv abstracts with tool access. - **Year**: 2026 - **Tasks**: 49 recent research-mathematics problems - **Format**: Final-answer accuracy with tools - **Difficulty**: Research mathematics ### ArXivMath Aug. 2026 (no tools) (ArXivMath August 2026 without tools) Final-answer research mathematics problems drawn from recent arXiv papers, including problems based on resolved or refuted conjectures. - **Year**: 2026 - **Tasks**: 57 recent research-mathematics problems - **Format**: Final-answer accuracy - **Difficulty**: Research mathematics ### ArXivMath Aug. 2026 (tools) (ArXivMath August 2026 with tools) The August 2026 ArXivMath release evaluated with a code-execution sandbox and no internet access. - **Year**: 2026 - **Tasks**: 57 recent research-mathematics problems - **Format**: Final-answer accuracy with tools - **Difficulty**: Research mathematics ### AA AIME 2025 (Artificial Analysis AIME 2025) An independently evaluated AIME 2025 result from Artificial Analysis. - **Year**: 2026 - **Tasks**: 30 AIME 2025 problems - **Format**: Accuracy - **Difficulty**: Olympiad mathematics ### AIME 2023 (American Invitational Mathematics Examination 2023) A 15-question, 3-hour examination where each answer is an integer from 000 to 999. Serves as the intermediate step between AMC 10/12 and the USA Mathematical Olympiad (USAMO). - **Year**: 2023 - **Tasks**: 15 problems - **Format**: Integer answers 000-999 - **Difficulty**: High school olympiad level ### AIME 2024 (American Invitational Mathematics Examination 2024) The 2024 edition of AIME, maintaining the same format of 15 challenging mathematics problems with integer answers from 000 to 999. - **Year**: 2024 - **Tasks**: 15 problems - **Format**: Integer answers 000-999 - **Difficulty**: High school olympiad level ### AIME 2025 (American Invitational Mathematics Examination 2025) The most recent AIME examination, featuring 15 challenging mathematics problems testing olympiad-level mathematical reasoning with integer answers from 000-999. - **Year**: 2025 - **Tasks**: 15 problems - **Format**: Integer answers 000-999 - **Difficulty**: High school olympiad level ### GSM8K (Grade School Math 8K) GSM8K tests multi-step arithmetic in grade-school word problems. OpenAI introduced the dataset in 2021; a high score on this fixed test is one piece of evidence about mathematical reasoning. - **Year**: 2021 - **Tasks**: Grade-school math word problems - **Format**: Exact match - **Difficulty**: Grade-school math ### MATH (MATH) A competition-style mathematical reasoning benchmark reported in DeepSeek-V4 base-model evaluations. - **Year**: 2026 - **Tasks**: Competition math problems - **Format**: Exact match - **Difficulty**: Advanced math reasoning ### CMath (CMath) A Chinese mathematical reasoning benchmark reported in DeepSeek-V4 base-model evaluations. - **Year**: 2026 - **Tasks**: Chinese math problems - **Format**: Exact match - **Difficulty**: Math reasoning ### AIME25 (Arcee) (AIME25 first-party comparison snapshot) A display-only AIME25 reference from Arcee AI's Trinity-Large-Thinking launch chart. - **Year**: 2026 - **Tasks**: 15 problems - **Format**: Integer answers 000-999 - **Difficulty**: High school olympiad level ### HMMT Feb 2023 (Harvard-MIT Mathematics Tournament February 2023) A prestigious high school mathematics competition hosted jointly by Harvard and MIT, featuring challenging problems across various mathematical disciplines. - **Year**: 2023 - **Tasks**: Tournament problems - **Format**: Competition mathematics - **Difficulty**: High school olympiad level ### HMMT Feb 2024 (Harvard-MIT Mathematics Tournament February 2024) The 2024 February edition of the Harvard-MIT Mathematics Tournament, continuing the tradition of challenging high school mathematics competition. - **Year**: 2024 - **Tasks**: Tournament problems - **Format**: Competition mathematics - **Difficulty**: High school olympiad level ### HMMT Feb 2025 (Harvard-MIT Mathematics Tournament February 2025) The most recent February edition of the Harvard-MIT Mathematics Tournament, featuring the latest challenging problems in competitive mathematics. - **Year**: 2025 - **Tasks**: Tournament problems - **Format**: Competition mathematics - **Difficulty**: High school olympiad level ### BRUMO 2025 (Bulgarian Mathematical Olympiad 2025) A challenging mathematical olympiad competition featuring problems that test advanced mathematical reasoning and problem-solving skills at the olympiad level. - **Year**: 2025 - **Tasks**: Olympiad problems - **Format**: Mathematical olympiad - **Difficulty**: Mathematical olympiad level ### MATH-500 (MATH-500 Problem Set) A curated subset of 500 problems from the MATH dataset, covering algebra, counting and probability, geometry, intermediate algebra, number theory, prealgebra, and precalculus. - **Year**: 2021 - **Tasks**: 500 problems - **Format**: Free-form mathematical answers - **Difficulty**: High school to undergraduate ### AIME26 (AIME 2026) A 2026 American Invitational Mathematics Examination snapshot used in frontier-model comparison tables for mathematical reasoning. - **Year**: 2026 - **Tasks**: Competition math problems - **Format**: Short-answer mathematics - **Difficulty**: Olympiad-style mathematics ### IPhO 2025 (Theory) (International Physics Olympiad 2025 (Theory)) The three official theory problems from the 2025 International Physics Olympiad, scored with blinded human evaluation. - **Year**: 2026 - **Tasks**: 3 olympiad theory problems - **Format**: Physics olympiad theory - **Difficulty**: International olympiad physics ### HMMT Feb 2025 (Harvard-MIT Mathematics Tournament February 2025) A February 2025 HMMT slice used in exact-value provider tables for advanced contest-math reasoning. - **Year**: 2025 - **Tasks**: Competition math problems - **Format**: Contest mathematics - **Difficulty**: Olympiad-style mathematics ### HMMT Nov 2025 (Harvard-MIT Mathematics Tournament November 2025) A November 2025 HMMT slice for high-end mathematical reasoning comparisons. - **Year**: 2025 - **Tasks**: Competition math problems - **Format**: Contest mathematics - **Difficulty**: Olympiad-style mathematics ### HMMT Feb 2026 (Harvard-MIT Mathematics Tournament February 2026) A February 2026 HMMT slice used in newer frontier-model math comparisons. - **Year**: 2026 - **Tasks**: Competition math problems - **Format**: Contest mathematics - **Difficulty**: Olympiad-style mathematics ### IMOAnswerBench (IMOAnswerBench) A challenging mathematical reasoning benchmark reported in DeepSeek-V4 model evaluations. - **Year**: 2026 - **Tasks**: Advanced mathematical answer generation - **Format**: Pass@1 math benchmark - **Difficulty**: Olympiad-level mathematics ### Apex (Apex) A high-difficulty mathematical reasoning benchmark reported in DeepSeek-V4 model evaluations. - **Year**: 2026 - **Tasks**: Advanced mathematical reasoning - **Format**: Pass@1 math benchmark - **Difficulty**: Frontier math reasoning ### Apex Shortlist (Apex Shortlist) A shortlist subset of the Apex mathematical reasoning benchmark reported in DeepSeek-V4 model evaluations. - **Year**: 2026 - **Tasks**: Advanced mathematical reasoning - **Format**: Pass@1 math benchmark - **Difficulty**: Frontier math reasoning ### MMAnswerBench (MMAnswerBench) A multimodal mathematical reasoning benchmark that tests whether models can answer visually grounded math questions correctly. - **Year**: 2026 - **Tasks**: Multimodal math questions - **Format**: Visual and structured mathematical QA - **Difficulty**: Advanced mathematical reasoning ### FrontierMath (legacy) (FrontierMath legacy aggregate) Legacy FrontierMath values retained for historical model pages. This field is not used in current rankings because it can mix prior benchmark versions and slices. - **Year**: 2024 - **Tasks**: Historical aggregate - **Format**: Open-ended mathematical reasoning with tool access - **Difficulty**: Research-level mathematics ### FrontierMath v2 (Tiers 1-3) (FrontierMath v2 Tiers 1-3) Epoch AI's corrected v2 core FrontierMath suite of private advanced mathematics problems. Models can reason iteratively and use Python; scores are pass rates on the private set. - **Year**: 2026 - **Tasks**: 285 private advanced mathematics problems - **Format**: Python-enabled iterative mathematical problem solving - **Difficulty**: From olympiad-plus to early research mathematics ### FrontierMath v2 (Tier 4) (FrontierMath v2 Tier 4) Epoch AI's corrected v2 Tier 4 expansion, a separate set of exceptionally difficult research-level mathematics problems evaluated with Python-enabled iterative reasoning. - **Year**: 2026 - **Tasks**: 41 private extreme-difficulty mathematics problems - **Format**: Python-enabled iterative mathematical problem solving - **Difficulty**: Research-level mathematics requiring hours or days of expert work ### USAMO 2026 (United States of America Mathematical Olympiad 2026) The premier US mathematical olympiad competition, featuring proof-based problems that require deep mathematical insight and rigorous argumentation at the highest competition level. - **Year**: 2026 - **Tasks**: 6 proof-based problems - **Format**: Mathematical proof construction - **Difficulty**: International olympiad level ### ProofBench (Vals ProofBench) Vals AI automated theorem-proving benchmark. - **Year**: 2026 - **Tasks**: Automated theorem proving - **Format**: Accuracy score - **Difficulty**: Formal proof reasoning ### AIME (Vals AIME) Challenging national math exam given to top high-school students - **Year**: 2026 - **Tasks**: AIME math problems - **Format**: Accuracy score - **Difficulty**: Competition math ### AIME 2025 (Mercor) (AIME 2025 — Mercor evaluation) Solving competition math problems with integer answers. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: 30 source-reported tasks - **Format**: Pass@1 - **Difficulty**: Source-specific evaluation ## Decision Models Benchmarks ### JevBench (JevBench) Typed-decision evaluation across 1,624 decisions, combining chance-corrected intelligence, probability calibration, latency, and cost. - **Year**: 2026 - **Tasks**: 904 open decisions plus 720 sealed decisions per complete run - **Format**: Official option A harmonic composite with published uncertainty - **Difficulty**: Choice, Noul, and Score decisions with sealed generalization tests ### Cloudflare Decision Index 0.2.1 (Cloudflare Decision Index 0.2.1) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: BPoMP (Cloudflare Decision Index 0.2.1: BPoMP) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: Humicroedit (Cloudflare Decision Index 0.2.1: Humicroedit) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: POP909-CL (Cloudflare Decision Index 0.2.1: POP909-CL) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: cfcolor (Cloudflare Decision Index 0.2.1: cfcolor) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: ForecastBench (Cloudflare Decision Index 0.2.1: ForecastBench) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: Habermas Machine (Cloudflare Decision Index 0.2.1: Habermas Machine) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: New Yorker (Cloudflare Decision Index 0.2.1: New Yorker) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: GPQA Diamond (Cloudflare Decision Index 0.2.1: GPQA Diamond) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: GSM8K (Cloudflare Decision Index 0.2.1: GSM8K) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: ChessBench (Cloudflare Decision Index 0.2.1: ChessBench) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: MuSR (Cloudflare Decision Index 0.2.1: MuSR) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: SATA-Bench (Cloudflare Decision Index 0.2.1: SATA-Bench) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: CRUXEval (Cloudflare Decision Index 0.2.1: CRUXEval) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: CLadder (Cloudflare Decision Index 0.2.1: CLadder) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: HLE (Cloudflare Decision Index 0.2.1: HLE) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: MMLU-Pro (Cloudflare Decision Index 0.2.1: MMLU-Pro) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: BBH (Cloudflare Decision Index 0.2.1: BBH) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: ContractNLI (Cloudflare Decision Index 0.2.1: ContractNLI) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: ANLI (Cloudflare Decision Index 0.2.1: ANLI) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: WinoGrande (Cloudflare Decision Index 0.2.1: WinoGrande) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: HellaSwag (Cloudflare Decision Index 0.2.1: HellaSwag) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: ACOS (Cloudflare Decision Index 0.2.1: ACOS) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: FinEntity (Cloudflare Decision Index 0.2.1: FinEntity) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: iSarcasmEval (Cloudflare Decision Index 0.2.1: iSarcasmEval) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: VAST (Cloudflare Decision Index 0.2.1: VAST) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: NLI4CT (Cloudflare Decision Index 0.2.1: NLI4CT) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: RAGTruth (Cloudflare Decision Index 0.2.1: RAGTruth) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: BANKING77 (Cloudflare Decision Index 0.2.1: BANKING77) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: CLINC150+OOS (Cloudflare Decision Index 0.2.1: CLINC150+OOS) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: BRIGHT (Cloudflare Decision Index 0.2.1: BRIGHT) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: Amazon ESCI (Cloudflare Decision Index 0.2.1: Amazon ESCI) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: PhishNChips phishing decisions (Cloudflare Decision Index 0.2.1: PhishNChips phishing decisions) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: HoVer (Cloudflare Decision Index 0.2.1: HoVer) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: BFCL (Cloudflare Decision Index 0.2.1: BFCL) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: ToolRet (Cloudflare Decision Index 0.2.1: ToolRet) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: API-Bank (Cloudflare Decision Index 0.2.1: API-Bank) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: Home appliance simulator (Cloudflare Decision Index 0.2.1: Home appliance simulator) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: When2Call (Cloudflare Decision Index 0.2.1: When2Call) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: RouterBench (Cloudflare Decision Index 0.2.1: RouterBench) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: SGD/SGD-X (Cloudflare Decision Index 0.2.1: SGD/SGD-X) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: MMLU (Cloudflare Decision Index 0.2.1: MMLU) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: ARC-Easy (Cloudflare Decision Index 0.2.1: ARC-Easy) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: ARC-Challenge (Cloudflare Decision Index 0.2.1: ARC-Challenge) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: OpenBookQA (Cloudflare Decision Index 0.2.1: OpenBookQA) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: CommonsenseQA (Cloudflare Decision Index 0.2.1: CommonsenseQA) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: Support-ticket calibration (Cloudflare Decision Index 0.2.1: Support-ticket calibration) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: Phishing difficulty gradient (Cloudflare Decision Index 0.2.1: Phishing difficulty gradient) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### Cloudflare Decision Index 0.2.1: Email spam classification (Cloudflare Decision Index 0.2.1: Email spam classification) Published Cloudflare and community decision-model configurations. - **Year**: 2026 ### FinancialPhraseBank (FinancialPhraseBank) Classify financial news sentences as positive, negative, or neutral. FinancialPhraseBank tests whether a model interprets financial context rather than treating every favorable-sounding word as positive. - **Year**: 2013 - **Tasks**: Financial sentiment classification - **Format**: Decision classification - **Difficulty**: Depends on the evaluated split and label mapping ### RAGTruth (RAGTruth) RAGTruth annotates unsupported or contradictory content in answers generated with retrieved evidence. It supports evaluation of hallucination detectors across question answering, summarization, and data-to-text tasks. - **Year**: 2024 - **Tasks**: Detecting unsupported claims in retrieved-context answers - **Format**: Decision classification - **Difficulty**: Depends on the evaluated split and label mapping ### JudgeBench (JudgeBench) JudgeBench evaluates whether a judge can identify the better response in pairs with objective correctness labels. It tests judgment on challenging questions rather than agreement with writing style preferences. - **Year**: 2024 - **Tasks**: Choosing the better of two model responses - **Format**: Decision classification - **Difficulty**: Depends on the evaluated split and label mapping ### TabFact (TabFact) TabFact asks whether a statement is supported or refuted by a Wikipedia table. It tests decisions that combine language interpretation with structured evidence. - **Year**: 2019 - **Tasks**: Fact verification against tables - **Format**: Decision classification - **Difficulty**: Depends on the evaluated split and label mapping ### ContractNLI (ContractNLI) ContractNLI asks whether a contract entails, contradicts, or does not mention a hypothesis. The original task also requires evidence spans that support the classification. - **Year**: 2021 - **Tasks**: Contract entailment and evidence identification - **Format**: Decision classification - **Difficulty**: Depends on the evaluated split and label mapping ### Circa (Circa) Circa pairs polar questions with indirect answers and labels how each answer should be interpreted. It tests conversational decisions where the response does not simply say yes or no. - **Year**: 2020 - **Tasks**: Interpreting indirect answers to yes/no questions - **Format**: Decision classification - **Difficulty**: Depends on the evaluated split and label mapping ### Belebele (Belebele) Belebele tests passage comprehension through questions with four answer options. Its parallel language variants let researchers study how reading performance changes across languages. - **Year**: 2023 - **Tasks**: Multilingual multiple-choice reading comprehension - **Format**: Decision classification - **Difficulty**: Depends on the evaluated split and label mapping ### WinoGrande (Perplexity panel) (WinoGrande (Perplexity panel)) Ambiguous-reference resolution on the 1,000-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup. - **Year**: 2026 - **Tasks**: Ambiguous-reference resolution - **Format**: Accuracy on the Perplexity fixed panel - **Difficulty**: Depends on the sampled tasks and label mapping ### FinancialPhraseBank (Perplexity panel) (FinancialPhraseBank (Perplexity panel)) Financial sentiment classification on the 999-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup. - **Year**: 2026 - **Tasks**: Financial sentiment classification - **Format**: Accuracy on the Perplexity fixed panel - **Difficulty**: Depends on the sampled tasks and label mapping ### RAGTruth (Perplexity panel) (RAGTruth (Perplexity panel)) Unsupported-claim detection on the 1,500-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup. - **Year**: 2026 - **Tasks**: Unsupported-claim detection - **Format**: Accuracy on the Perplexity fixed panel - **Difficulty**: Depends on the sampled tasks and label mapping ### JudgeBench (Perplexity panel) (JudgeBench (Perplexity panel)) Response-pair judgment on the 350-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup. - **Year**: 2026 - **Tasks**: Response-pair judgment - **Format**: Accuracy on the Perplexity fixed panel - **Difficulty**: Depends on the sampled tasks and label mapping ### BBH (Perplexity panel) (BBH (Perplexity panel)) Challenging reasoning tasks on the 750-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup. - **Year**: 2026 - **Tasks**: Challenging reasoning tasks - **Format**: Accuracy on the Perplexity fixed panel - **Difficulty**: Depends on the sampled tasks and label mapping ### JevBench public hard (Perplexity panel) (JevBench public hard (Perplexity panel)) Public-hard decision accuracy on the 101-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup. - **Year**: 2026 - **Tasks**: Public-hard decision accuracy - **Format**: Accuracy on the Perplexity fixed panel - **Difficulty**: Depends on the sampled tasks and label mapping ### TabFact (Perplexity panel) (TabFact (Perplexity panel)) Table-based fact verification on the 500-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup. - **Year**: 2026 - **Tasks**: Table-based fact verification - **Format**: Accuracy on the Perplexity fixed panel - **Difficulty**: Depends on the sampled tasks and label mapping ### ContractNLI (Perplexity panel) (ContractNLI (Perplexity panel)) Contract hypothesis classification on the 510-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup. - **Year**: 2026 - **Tasks**: Contract hypothesis classification - **Format**: Accuracy on the Perplexity fixed panel - **Difficulty**: Depends on the sampled tasks and label mapping ### Circa (Perplexity panel) (Circa (Perplexity panel)) Indirect yes/no answer interpretation on the 500-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup. - **Year**: 2026 - **Tasks**: Indirect yes/no answer interpretation - **Format**: Accuracy on the Perplexity fixed panel - **Difficulty**: Depends on the sampled tasks and label mapping ### Belebele (Perplexity panel) (Belebele (Perplexity panel)) Passage comprehension on the 500-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup. - **Year**: 2026 - **Tasks**: Passage comprehension - **Format**: Accuracy on the Perplexity fixed panel - **Difficulty**: Depends on the sampled tasks and label mapping ### TruthfulQA binary (Perplexity panel) (TruthfulQA binary (Perplexity panel)) Binary truthfulness classification on the 500-row sample in Perplexity's September 2026 decision panel. This is a sampled, provider-reported accuracy result with its own evaluation setup. - **Year**: 2026 - **Tasks**: Binary truthfulness classification - **Format**: Accuracy on the Perplexity fixed panel - **Difficulty**: Depends on the sampled tasks and label mapping ### Perplexity Decision Panel (Perplexity Decision Panel) Perplexity reports accuracy across 11 decision tasks on a fixed 7,210-row panel. The overall percentage combines unequal task sample sizes, so a larger task contributes more than a smaller task. - **Year**: 2026 - **Tasks**: 11 decision task families - **Format**: Accuracy on the Perplexity fixed panel - **Difficulty**: Depends on the sampled tasks and label mapping ### Decision Index 0.2.1 (Fastino report) (Decision Index 0.2.1 (Fastino report)) Fastino reports GLiDE at 64.81 skill points on Decision Index 0.2.1 in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference. - **Year**: 2026 - **Tasks**: 38 component benchmarks - **Format**: Chance-adjusted skill points - **Difficulty**: Varies by task and supplied answer options ### Knowledge and Reasoning (Fastino report) (Knowledge and Reasoning (Fastino report)) Fastino reports GLiDE at 62.9 skill points on Knowledge and Reasoning in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference. - **Year**: 2026 - **Tasks**: 10 component benchmarks - **Format**: Chance-adjusted skill points - **Difficulty**: Varies by task and supplied answer options ### Language Understanding (Fastino report) (Language Understanding (Fastino report)) Fastino reports GLiDE at 63.8 skill points on Language Understanding in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference. - **Year**: 2026 - **Tasks**: 10 component benchmarks - **Format**: Chance-adjusted skill points - **Difficulty**: Varies by task and supplied answer options ### Retrieval and Classification (Fastino report) (Retrieval and Classification (Fastino report)) Fastino reports GLiDE at 60.9 skill points on Retrieval and Classification in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference. - **Year**: 2026 - **Tasks**: 6 component benchmarks - **Format**: Chance-adjusted skill points - **Difficulty**: Varies by task and supplied answer options ### Tools and Automation (Fastino report) (Tools and Automation (Fastino report)) Fastino reports GLiDE at 83.5 skill points on Tools and Automation in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference. - **Year**: 2026 - **Tasks**: 5 component benchmarks - **Format**: Chance-adjusted skill points - **Difficulty**: Varies by task and supplied answer options ### Arts and Human Taste (Fastino report) (Arts and Human Taste (Fastino report)) Fastino reports GLiDE at 46.0 skill points on Arts and Human Taste in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference. - **Year**: 2026 - **Tasks**: 7 component benchmarks - **Format**: Chance-adjusted skill points - **Difficulty**: Varies by task and supplied answer options ### CLadder (Fastino report) (CLadder (Fastino report)) Fastino reports GLiDE at 88.7% on CLadder in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference. - **Year**: 2026 - **Tasks**: Decision Index task conversion - **Format**: Raw accuracy (%) - **Difficulty**: Varies by task and supplied answer options ### CRUXEval (Fastino report) (CRUXEval (Fastino report)) Fastino reports GLiDE at 92.6% on CRUXEval in its Decision Index 0.2.1 run. The comparison quotes the published Jev 1.13.0 reference. - **Year**: 2026 - **Tasks**: Decision Index task conversion - **Format**: Raw accuracy (%) - **Difficulty**: Varies by task and supplied answer options ### Decision Index 0.3 Full Score (Decision Index 0.3) Decision Index 0.3 publishes full scores for 112 evaluated systems. The composite weights 37 public benchmarks at 20%, private same-skill tests at 50%, and private new-domain tasks at 30%. - **Year**: 2026 - **Tasks**: 110,201 decisions across 37 public benchmarks and private task sets - **Format**: Chance-corrected composite score - **Difficulty**: Mixed public and private decision tasks ### Decision Index 0.3 Knowledge (Decision Index 0.3 Knowledge) Decision Index 0.3 publishes chance-corrected knowledge scores for 112 evaluated systems. - **Year**: 2026 - **Tasks**: Knowledge area of the Decision Index 0.3 evaluation - **Format**: Chance-corrected area score - **Difficulty**: Knowledge decision tasks ### Decision Index 0.3 Language (Decision Index 0.3 Language) Decision Index 0.3 publishes chance-corrected language scores for 112 evaluated systems. - **Year**: 2026 - **Tasks**: Language area of the Decision Index 0.3 evaluation - **Format**: Chance-corrected area score - **Difficulty**: Language decision tasks ### Decision Index 0.3 Retrieval (Decision Index 0.3 Retrieval) Decision Index 0.3 publishes chance-corrected retrieval scores for 112 evaluated systems. - **Year**: 2026 - **Tasks**: Retrieval area of the Decision Index 0.3 evaluation - **Format**: Chance-corrected area score - **Difficulty**: Retrieval and classification decision tasks ### Decision Index 0.3 Tools (Decision Index 0.3 Tools) Decision Index 0.3 publishes chance-corrected tools scores for 112 evaluated systems. - **Year**: 2026 - **Tasks**: Tools area of the Decision Index 0.3 evaluation - **Format**: Chance-corrected area score - **Difficulty**: Tool-use and automation decision tasks ### Decision Index 0.3 Arts (Decision Index 0.3 Arts) Decision Index 0.3 publishes chance-corrected arts scores for 112 evaluated systems. - **Year**: 2026 - **Tasks**: Arts area of the Decision Index 0.3 evaluation - **Format**: Chance-corrected area score - **Difficulty**: Arts and human-taste decision tasks ### Decision Index 0.2.1 (Liquid report) (Decision Index 0.2.1 (Liquid report)) Liquid AI reports 48.57 skill points for d1-3B and 15.95 for d1-omni-600M on Decision Index 0.2.1 with the official Decision Index 0.2.1 scorer. - **Year**: 2026 - **Format**: Chance-adjusted skill points - **Difficulty**: Varies by task and supplied answer options ### Knowledge (Liquid report) (Knowledge (Liquid report)) Liquid AI reports 23.8 skill points for d1-3B and 8.3 for d1-omni-600M on Knowledge with the official Decision Index 0.2.1 scorer. - **Year**: 2026 - **Format**: Chance-adjusted skill points - **Difficulty**: Varies by task and supplied answer options ### Language (Liquid report) (Language (Liquid report)) Liquid AI reports 56.4 skill points for d1-3B and 12.9 for d1-omni-600M on Language with the official Decision Index 0.2.1 scorer. - **Year**: 2026 - **Format**: Chance-adjusted skill points - **Difficulty**: Varies by task and supplied answer options ### Retrieval (Liquid report) (Retrieval (Liquid report)) Liquid AI reports 52.8 skill points for d1-3B and 35.0 for d1-omni-600M on Retrieval with the official Decision Index 0.2.1 scorer. - **Year**: 2026 - **Format**: Chance-adjusted skill points - **Difficulty**: Varies by task and supplied answer options ### Tools (Liquid report) (Tools (Liquid report)) Liquid AI reports 74.5 skill points for d1-3B and 15.1 for d1-omni-600M on Tools with the official Decision Index 0.2.1 scorer. - **Year**: 2026 - **Format**: Chance-adjusted skill points - **Difficulty**: Varies by task and supplied answer options ### Arts (Liquid report) (Arts (Liquid report)) Liquid AI reports 36.3 skill points for d1-3B and 6.8 for d1-omni-600M on Arts with the official Decision Index 0.2.1 scorer. - **Year**: 2026 - **Format**: Chance-adjusted skill points - **Difficulty**: Varies by task and supplied answer options ### Artificial Analysis Decision Index (Artificial Analysis Decision Index) Artificial Analysis scores decision models on its Decision Index v1.0 and runs every evaluation itself. The index is the weighted average of AA-Omniscience-Decision accuracy and non-hallucination rate (12.5% each), HLE-MultiChoice, When2Call and WildGuardTest (25% each). - **Year**: 2026 - **Format**: Weighted average, 0 to 100 - **Difficulty**: Weighted average of four decision-model evaluations ### AA-Omniscience-Decision accuracy (AA-Omniscience-Decision accuracy) AA-Omniscience-Decision accuracy is one component of the Artificial Analysis Decision Index v1.0, run by Artificial Analysis. It is the share of 500 private knowledge questions answered correctly, each with four answers and an "I don't know" option. - **Year**: 2026 - **Format**: Accuracy - **Difficulty**: Knowledge questions with an abstain option ### AA-Omniscience-Decision non-hallucination rate (AA-Omniscience-Decision non-hallucination rate) AA-Omniscience-Decision non-hallucination rate is one component of the Artificial Analysis Decision Index v1.0, run by Artificial Analysis. It is the share of questions not answered correctly where the model chose "I don't know" instead of a wrong answer. Higher is better. - **Year**: 2026 - **Format**: Abstention rate on missed questions - **Difficulty**: Knowledge questions with an abstain option ### HLE-MultiChoice (Decision Index) (HLE-MultiChoice (Decision Index)) HLE-MultiChoice is one component of the Artificial Analysis Decision Index v1.0, run by Artificial Analysis. It measures accuracy on the 513 text-only multiple-choice Humanity's Last Exam questions, each posed as one choice among lettered answers. - **Year**: 2026 - **Format**: Accuracy - **Difficulty**: Expert-level multiple choice ### When2Call (Decision Index) (When2Call (Decision Index)) When2Call is one component of the Artificial Analysis Decision Index v1.0, run by Artificial Analysis. It measures accuracy on NVIDIA's tool-calling evaluation: given a conversation and tools, choose to answer directly, call a tool, ask for more information, or decline. - **Year**: 2026 - **Format**: Accuracy - **Difficulty**: Tool-use decisions ### WildGuardTest (Decision Index) (WildGuardTest (Decision Index)) WildGuardTest is one component of the Artificial Analysis Decision Index v1.0, run by Artificial Analysis. It measures average F1 on Ai2's safety classification set: whether a request is harmful, whether a reply is a refusal, and whether that reply is harmful. - **Year**: 2026 - **Format**: Average F1 - **Difficulty**: Safety classification ## Korean Benchmarks ### KMMLU (Korean Massive Multitask Language Understanding) Evaluates Korean expert-level knowledge across 45 subjects. 20% of questions require Korean cultural context. - **Year**: 2024 - **Tasks**: 35,030 questions - **Format**: Multiple choice questions - **Difficulty**: Elementary to professional level in Korean ### KMMLU-Hard (KMMLU-Hard) A filtered hard subset of KMMLU containing ~5,000 questions that most models get wrong. - **Year**: 2025 - **Tasks**: ~5,000 questions - **Format**: Multiple choice questions - **Difficulty**: Advanced Korean reasoning ### KMMLU-Redux (KMMLU-Redux) Cleaned KMMLU from national technical qualification exams, with errors removed, decontaminated, and deduplicated. - **Tasks**: ~3,500 questions - **Format**: Technical multiple choice - **Difficulty**: Industrial/technical ### KMMLU-Pro (KMMLU-Pro) Korean National Professional Licensure exams evaluating professional-grade knowledge. - **Tasks**: ~2,500 questions - **Format**: Professional licensure exams - **Difficulty**: Professional ### CLIcK (Cultural and Linguistic Intelligence in Korean) Evaluates Korean culture and linguistics. - **Tasks**: 1,995 questions - **Format**: Cultural/linguistic QA - **Difficulty**: Korean cultural nuances ### KoBALT (Korean Benchmark for Advanced Linguistic Tasks) Evaluates advanced Korean linguistic competence. - **Tasks**: Linguistics questions - **Format**: Advanced linguistics - **Difficulty**: Advanced linguistic phenomena ### Korean CSAT (College Scholastic Ability Test (수능)) The Korean SAT exam. - **Tasks**: Multi-subject exam - **Format**: Standardized test - **Difficulty**: High school to college level ### HRM8K (HAE-RAE Math 8K) Korean mathematical reasoning (high-school to Olympiad level). - **Tasks**: 8,011 instances - **Format**: Math word problems - **Difficulty**: Olympiad level ## Cross-domain Benchmarks ### DistressBench (Scale Labs DistressBench) Measures crisis-support replies against clinician-written rubrics. These published results are excluded from overall and category rankings. - **Year**: 2026 - **Tasks**: 718 conversations (359 public, 359 private) - **Format**: Weighted rubric score (%) - **Difficulty**: Crisis-support response evaluation ### Mental Health Bench (Vals) (Vals Mental Health Bench) Published chatbot failure rates across urgent safety, health advice, and relationship boundaries in simulated teen conversations. - **Year**: 2026 - **Tasks**: Simulated teen conversations across three risk areas - **Format**: Urgent safety failure rate - **Difficulty**: Conversation safety evaluation ### LiveBench (LiveBench) A frequently refreshed benchmark with objective scoring across reasoning, coding, agentic coding, mathematics, data analysis, language, and instruction following. - **Year**: 2024 - **Tasks**: 23 objective tasks across 7 categories - **Format**: Mean of category averages - **Difficulty**: Broad frontier-model evaluation ### Vals Index (Vals Index v2) Vals AI composite benchmark across professional finance, coding, modeling, and legal-work tasks, including Finance Agent v2, EMB, Terminal-Bench 2.1, Vibe Code Bench, Code Migration, Legal Research, and HLAB. - **Year**: 2026 - **Tasks**: Finance, coding, spreadsheet modeling, code migration, and legal-work components - **Format**: Composite score - **Difficulty**: Private economic-work benchmark composite ### CAIS Text Leaderboard (CAIS AI Dashboard Text Capabilities Index) A Center for AI Safety dashboard view summarizing text capabilities across HLE and TextQuests. - **Year**: 2025 - **Tasks**: HLE and TextQuests - **Format**: Average component score - **Difficulty**: Composite frontier text capability ### APEX-1 (Mercor) (APEX-1 — Mercor evaluation) Single-turn professional deliverables graded against expert-written criteria. This table shows Mercor-run configurations for reference and is excluded from model rankings. - **Tasks**: Task and run counts differ across source pages - **Format**: Mean Score - **Difficulty**: Source-specific evaluation ### Cohere Embed 5 launch: ViDoRe V3 parsed-text average (Cohere Embed 5 launch: ViDoRe V3 parsed-text average) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: FinanceBench text retrieval (Cohere Embed 5 launch: FinanceBench text retrieval) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: FinQA text retrieval (Cohere Embed 5 launch: FinQA text retrieval) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: ViDoRe V3 Finance (Cohere Embed 5 launch: ViDoRe V3 Finance) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Parsed PDF full-suite average (Cohere Embed 5 launch: Parsed PDF full-suite average) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Fused text-image average (Cohere Embed 5 launch: Fused text-image average) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Page-image finance average (Cohere Embed 5 launch: Page-image finance average) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: European-language composite average (Cohere Embed 5 launch: European-language composite average) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Japanese retrieval composite (Cohere Embed 5 launch: Japanese retrieval composite) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Chinese retrieval composite (Cohere Embed 5 launch: Chinese retrieval composite) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Korean retrieval composite (Cohere Embed 5 launch: Korean retrieval composite) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Arabic retrieval composite (Cohere Embed 5 launch: Arabic retrieval composite) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Farsi retrieval composite (Cohere Embed 5 launch: Farsi retrieval composite) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Hindi retrieval composite (Cohere Embed 5 launch: Hindi retrieval composite) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Bengali retrieval composite (Cohere Embed 5 launch: Bengali retrieval composite) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Telugu retrieval composite (Cohere Embed 5 launch: Telugu retrieval composite) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Indonesian retrieval composite (Cohere Embed 5 launch: Indonesian retrieval composite) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Thai retrieval composite (Cohere Embed 5 launch: Thai retrieval composite) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Document throughput (Cohere Embed 5 launch: Document throughput) Cohere launch retrieval report. - **Year**: 2026 ### Cohere Embed 5 launch: Shared-space corpus/query pairings (Cohere Embed 5 launch: Shared-space corpus/query pairings) Cohere launch retrieval report. - **Year**: 2026 Canonical page: https://benchlm.ai/benchmarks