AI Benchmarks Directory

Tests whether a system can discover a programming language's hidden rules through experiments and apply them to unseen tasks.

70 held-out program-synthesis tasks; 31 discovery targetsHeld-out accuracy (Best@3)Discovering and applying unfamiliar executable rules

Tests whether a system can discover altered logic rules and use them to prove or correctly decline unseen theorems.

70 held-out theorems; 24 altered rulesHeld-out accuracy (Best@3)Learning and applying unfamiliar inference rules

Google-run scientific workflow evaluation using a sixfold verifier timeout to address verification timeouts.

70 expert-curated scientific research workflowsResolution rate across 3 trials per taskFrontier scientific research workflows

Defensive vulnerability patching on 120 held-out tasks spanning 73 weakness types.

Audit and patch held-out C/C++, Go, Java, TypeScript/JavaScript, Python, and Rust repositoriesPass@1Frontier agent evaluation

Agentic data-analysis tasks scored against per-task rubrics at a 980K-token budget.

Agentic data research and analysis tasksNormalized rubric scoreProfessional data analysis

Average coverage of required claims in answers produced during real-world MCP tool-use workflows.

Production-like multi-server MCP workflowsMean claim coverageReal-world tool use

Strict task success requiring every expert-written legal-work rubric criterion to pass.

1,235 legal-agent tasksAll-criteria pass rateProfessional legal work

Mean fraction of expert-written rubric criteria passed across legal-agent tasks.

1,235 legal-agent tasksMean criterion-pass rateProfessional legal work

Harvey AI's strict held-out task success rate requiring every rubric criterion to pass.

Harvey-held-out legal-agent tasksAll-criteria pass rateProfessional legal work

Harvey AI's mean criterion-level score on its held-out legal-agent evaluation.

Harvey-held-out legal-agent tasksMean criterion-pass rateProfessional legal work

Fraction of Toolathlon Verified tasks solved in at least one of three trials.

108 verified real-world tool-use tasksPass@3Long-horizon application tool use

Fraction of Toolathlon Verified tasks solved in all three independent trials.

108 verified real-world tool-use tasksAll-three-trials pass rateLong-horizon application tool use

Average assistant turns per Toolathlon Verified trajectory.

108 verified real-world tool-use tasksAverage trajectory lengthLong-horizon application tool use

A continuously maintained benchmark for difficult computer work, including coding, deep learning, finance, engineering, math, and science tasks.

74 professional computer-work tasks across 7 domainsTask completion rateFrontier autonomous knowledge work

The current Terminal-Bench release measures difficult computer work after recalibrating task resources, fixing unstable tasks, and removing tasks that no longer separate frontier systems.

66 professional computer-work tasksTask completion rate across 5 trials per taskFrontier autonomous knowledge work

A benchmark of AI agents completing expert-curated research workflows across the life, physical, Earth, mathematical, and engineering sciences.

70 expert-curated scientific research workflowsResolution rate across 3 trials per taskFrontier scientific research workflows

A multi-turn benchmark of emotional and social intelligence using synthetic personas and pairwise LLM judging.

Up to 120 multi-turn persona scenarios per modelPairwise EloEmotional and social intelligence

A Markov Studios computer-use benchmark that asks agents to produce 2D drawings and 3D models in AutoCAD.

21 2D drawing tasks and 29 3D modeling tasksTask completion rate at a 75-point rubric thresholdProfessional CAD computer use

A display-only Elo rating from blinded comparisons of multi-file web applications built by coding agents.

Multi-file web application developmentElo from blinded human preferencesAgentic frontend development

An independently evaluated professional-work benchmark reported as Elo.

Professional knowledge-work tasksEloProfessional work

An independently evaluated automation benchmark from Artificial Analysis.

Business-process automation tasksTask success rateAgentic automation

An independently evaluated enterprise-operations benchmark from Artificial Analysis.

Enterprise operations workflowsTask success rateEnterprise agent operations

Archived v1.0 legal-agent results from Artificial Analysis, kept separate from v1.1.

Legal agent tasksPublished v1.0 scoreProfessional legal work

Legal-agent task success requiring every rubric criterion to pass and no material hallucination, averaged across three judges.

120 private legal tasks across 24 practice areasHallucination-Gated All-Pass RateProfessional legal work

The share of legal-deliverable rubric criteria passed, pooled across three judges before the hallucination gate.

120 private legal tasks across 24 practice areasCriterion Pass RateProfessional legal work

An independently evaluated IT-operations benchmark from Artificial Analysis.

IT incident-response tasksTask success rateEnterprise IT operations

An independently evaluated Tau3 banking benchmark from Artificial Analysis.

Banking tool-use workflowsTask success rateAgentic banking workflows

A benchmark for agentic software engineering tasks executed in real terminal environments. Models must inspect files, run commands, edit code, and recover from errors over multi-step workflows.

Terminal-based software tasksInteractive CLI agent evaluationProfessional software engineering

Terminal-Bench 2.1 results, mostly provider-reported, stored separately from the Terminal-Bench 2.0 lane.

Terminal-based software-agent tasksInteractive task success rateProfessional software engineering

BrowseComp tests whether a browsing agent can find difficult, fact-seeking answers across the web. Its 1,266 problems use short answers that are easier to grade than an open-ended research report.

Research questions requiring browsingWeb search and evidence synthesisHard web research

Tool-augmented Humanity's Last Exam scores reported in DeepSeek-V4 thinking-mode evaluations.

Expert questions with tool usePass@1Frontier tool-augmented reasoning

An agentic real-world work-task evaluation reported as an Elo score in DeepSeek-V4 thinking-mode evaluations.

Agentic real-world work tasksEloProfessional agentic workflows

An Artificial Analysis normalized score for economically valuable tasks.

Economically valuable tasksNormalized scoreProfessional agentic workflows

A display-only Artificial Analysis agentic index.

Cross-benchmark agentic indexAggregated model scoreDisplay-only external reference

Artificial Analysis' document-output benchmark: professional tasks whose deliverable is a produced PDF, one of the ten components of its Intelligence Index v4.3.

Professional document-production tasksTask success rateProfessional knowledge work

Artificial Analysis' data-analysis benchmark, testing agents on spreadsheet and document work to answer the quantitative questions a business or data analyst faces day to day.

Spreadsheet and document analysis questionsTask success rateBusiness and data analysis

Artificial Analysis' implementation of the APEX-Agents benchmark for long-horizon professional-services agent tasks.

452 professional-services agent tasksPass@1Long-horizon workplace agent tasks

A game-environment benchmark that evaluates AI models in novel games covering strategic planning, resource management, spatial reasoning, cooperation, and theory of mind.

Novel game environmentsComposite game leaderboardAgentic coding and decision-making

OSWorld-Verified is the July 2025 repaired release of OSWorld's real-computer evaluation. It measures whether a model-agent system can finish desktop and web tasks from configured starting states, with success checked by execution-based evaluators.

369 real-world computer tasks (361 when eight Google Drive tasks are excluded)Execution-based interactive task successMulti-step desktop and cross-application workflows

OSWorld 2.0 evaluates long-horizon computer-use tasks. Published results use different task releases, metrics, and run settings, so scores appear by comparable protocol and unmatched rows stay unranked.

108 long-horizon computer-use workflowsInteractive computer-use evaluation; binary completion and partial-score metricsLong-horizon professional workflows

A cybersecurity task benchmark for evaluating defensive cyber workflows and vulnerability-oriented agent performance.

1,507 vulnerability analysis instancesVulnerability reproduction and PoC generationReal-world cybersecurity

An external benchmark for evaluating whether coding agents can produce correct patches for real-world software vulnerabilities.

Real-world vulnerability patching tasksPass@1Automated vulnerability remediation

A cybersecurity benchmark that measures whether an agent can turn raw threat-intelligence reports into working detection rules.

Threat-intelligence-to-detection-rule workflowsSuccess rateProfessional cyber threat detection

A cybersecurity benchmark of professional Capture the Flag tasks for measuring autonomous cyber agent capability and risk.

40 professional CTF tasksCybersecurity agent task completionProfessional cybersecurity

A controlled benchmark for evaluating whether AI agents can extend vulnerability-triggering inputs into working exploits.

898 exploitation tasksWorking exploit generationAdvanced cybersecurity exploitation

An occupational agent benchmark for professional workflows that workers say they most want delegated to AI.

130 tasks across 35 occupationsAgentic workplace deliverablesProfessional multi-source workflows

A vision-language browsing benchmark for multimodal web research and tool-use workflows.

Multimodal browsing tasksVision-language web research evaluationMultimodal browser-agent

A computer-use benchmark for GUI task completion across the broader OSWorld task suite.

Computer-use tasksInteractive GUI evaluationBroad computer-use suite

A mobile GUI agent benchmark for completing Android app workflows and on-device tasks.

Android app workflowsInteractive mobile-agent evaluationComplex mobile task completion

A browser-agent benchmark for completing multi-step workflows on live websites.

Live website workflowsInteractive browser-agent evaluationMulti-step web navigation

A benchmark for tool-calling over Model Context Protocol integrations and external tools.

Tool-integrated agent tasksInteractive tool-calling evaluationAdvanced tool use

A Moonshot AI internal long-horizon agent benchmark for persistent professional coworking tasks.

17 professional scenarios, 610 evaluation pointsAverage pass rate across repeated OpenClaw runsLong-horizon agentic work

A human-verified edition of MCPMark for MCP tool use across Notion, GitHub, Filesystem, Postgres, and Playwright server environments.

MCP tool-use tasks across five server environmentsInteractive MCP task completionAdvanced tool use

A tool-use benchmark focused on selecting, sequencing, and completing tasks with external tools.

Multi-tool workflowsInteractive tool-calling evaluationAdvanced tool use

A verified tool-use benchmark variant for completing multi-step workflows with external tools.

Verified multi-tool workflowsInteractive tool-use scoreAdvanced tool use

An agent benchmark for completing automation workflows in reproducible task environments.

600 public automation tasksAgent task-completion scoreLong-horizon automation

An agent benchmark for completing business workflows across simulated applications and API endpoints.

600+ held-out workflows across 47 simulated applications and six business domainsTask pass rateLong-horizon automation

An agent benchmark reported in DeepSeek's V4 Flash 0731 launch comparison.

Agent tasksProvider-reported task scoreAdvanced agentic work

A professional-services agent benchmark covering long-horizon knowledge-work tasks.

Professional-services agent tasksAgent task-completion scoreLong-horizon professional work

A spreadsheet-focused benchmark for agentic analysis and editing workflows.

Spreadsheet analysis and editing tasksAgent task-completion scoreProfessional spreadsheet work

Moonshot AI's internal benchmark for presentation and deck-production workflows.

Internal presentation workflowsInternal evaluation scoreProfessional presentation creation

A Z.AI benchmark for OpenClaw-style agent workflows spanning information search, office work, data analysis, development and operations, automation, and security.

OpenClaw agent workflowsEnd-to-end agent benchmarkBroad productivity and operations workflows

This route is a sourced ledger for published τ²-bench results. Most current rows come from Artificial Analysis's telecom implementation, while named provider rows can use telecom, airline, retail, or aggregate setups.

Airline, retail, and telecom customer-service task setsPublished domain success or pass^k resultsDual-control customer-service workflows

An agentic browsing benchmark where models search the web, gather evidence, and answer list-style questions using browser tools.

Agentic browsing and list-answer questionsSearch / open / find browser-agent evaluationAgentic web research

τ²-bench Airline tests conversational agents on airline customer-service tasks governed by domain policy and database-changing tools.

Airline customer-service tasksDomain success under a published trial policyPolicy-constrained airline support workflows

An OpenClaw agent benchmark from Kilo that measures successful task completion across standardized real-world agent workflows.

23 OpenClaw agent tasksAverage success rate from official runsLong-horizon agent workflows

A holistic coding-agent benchmark that evaluates AI agents across issue resolution, frontend work, greenfield development, testing, and information gathering.

SWE-bench Verified, SWE-bench Multimodal, Commit0, SWT-bench Verified, and GAIAMacro-average across five coding-agent categoriesReal-world software engineering agent tasks

A Scale SWE-Atlas software-engineering agent benchmark focused on refactoring tasks.

SWE-Atlas refactoring tasksRefactoring score with confidence intervalsReal-world software-engineering agent tasks

Tests whether coding agents can find and fix multiple bugs across a repository without issue descriptions, while preserving existing behavior.

4,068 reference bugs across 100 repositories in 22 languagesMicro-average percentage of bugs resolved with a repository regression gateRepository-wide bug discovery and repair

Tests whether coding agents can complete long-horizon, whole-repository stack migrations while preserving the original program's behavior.

20 whole-repository stack migrationsMigration audit, frozen behavioral checks, and agentic verification6- to 30-hour autonomous repository migrations

Tests whether coding agents can improve the training algorithm inside an existing AI research codebase, then survive a sealed training run and held-out evaluation.

10 AI training-algorithm design tasksFour-hour code rewrite followed by sealed training and held-out evaluationEnd-to-end AI research and algorithm design

A benchmark for open-ended LLM inference optimization by AI agents. Agents receive a base model, one H100, and a fixed time budget to build a valid OpenAI-compatible inference server that improves serving speed.

4 inference-serving optimization scenariosTwo-hour autonomous CLI agent runOpen-ended ML systems engineering

A ByteDance Seed benchmark of 134 real-world, day-scale tasks that measures how autonomous agents learn from environment feedback over 12+ hour interaction horizons, spanning scientific and ML, systems and software engineering, optimization, knowledge, formal, and game domains.

134 tasks (51 public) across 6 domainsLong-horizon interactive agent evaluationDay-scale expert tasks

A function-calling benchmark for tool selection, schema adherence, and argument correctness.

Function-calling tasksTool invocation and schema evaluationAdvanced tool use

A lightweight machine-learning competition benchmark that measures whether models can iteratively train, evaluate, and improve ML systems in low-resource settings.

Low-resource ML competitionsAutonomous iterative ML optimizationAgentic machine learning

An OpenClaw-derived agent benchmark covering practical work and life tasks such as office document delivery, research, planning, and code maintenance.

OpenClaw-style real-world tasksAgent workflow evaluationBroad real-world agentic execution

A transparent real-world autonomous-agent benchmark with 300 human-verified tasks, 2,159 rubric items, and Pass^3 scoring across general, multi-turn, and native multimodal agent tasks.

300 tasks, 2,159 rubricsEnd-to-end autonomous-agent evaluation with Pass^3 scoringReal-world general, multi-turn, and native multimodal agent execution

An end-to-end autonomous scientific research benchmark with 40 tasks across 10 scientific domains, where agents receive related literature and raw data, then attempt to rediscover the hidden target paper.

40 tasks across 10 scientific domainsEnd-to-end autonomous research evaluation with RADS scoringScientific research re-discovery

Qwen's internal OpenClaw-style benchmark for measuring broad real-world agent performance across practical productivity and research tasks.

Real-world agent workflowsEnd-to-end agent evaluationBroad real-world agentic execution

A Qwen benchmark for artifact and webpage generation quality reported as an Elo-style rating.

Web artifacts and interactive deliverablesElo-style artifact benchmarkArtifact generation

τ³-bench is the current evolution of Sierra's tool-agent-user framework, adding corrected task releases and newer knowledge and voice evaluation modes alongside airline, retail, and telecom.

Corrected customer-service tasks plus knowledge and voice evaluation modesPublished domain or average success resultsLong-horizon, multimodal, and knowledge-aware tool use

An interactive real-world agent benchmark grounded in practical consumer-service tasks such as delivery, in-store consumption, and online travel workflows.

Interactive consumer-service agent tasksEnd-to-end interactive agent evaluationLong-horizon real-world workflows

A long-horizon planning benchmark that tests whether agents can optimize under explicit time, budget, and feasibility constraints.

Travel planning and constrained shoppingLong-horizon planning benchmarkConstrained agent planning

A Model Context Protocol task benchmark used in Qwen's launch tables to measure practical execution over MCP-style tools and integrations.

MCP-integrated tool tasksInteractive tool-use evaluationAdvanced MCP workflows

A broad research-agent benchmark for open-ended information gathering, synthesis, and answer construction across wide search spaces.

Open-ended research tasksMulti-source research evaluationBroad research-agent workflows

Qwen's internal benchmark for long-horizon professional work across computer science, finance, law, medicine, and other productivity domains.

Long-horizon professional workflowsProvider-run agent scoreCross-domain professional work

A mobile-use agent benchmark for completing interactive tasks in smartphone environments.

Interactive mobile-device workflowsMobile agent task scoreLong-horizon mobile computer use

GAIA evaluates AI models on real-world tasks that are conceptually simple for humans but require multi-step reasoning, web browsing, tool use, and multimodal understanding for AI. Tasks span three difficulty levels and test practical assistant capabilities rather than academic knowledge.

466

Original TAU-bench tests an agent's tool use and policy compliance in simulated airline and retail conversations. Its pass^k metric asks whether the agent succeeds in every one of k repeated trials.

Airline and retail task sets in the archived 2024 releaseDomain-specific pass^1 through pass^4 task successPolicy-constrained, multi-turn customer service

WebArena tests whether a browser-agent system can complete 812 long-horizon tasks inside self-hosted replicas of functional websites. It checks the requested end state, so a result reflects the model, agent scaffold, browser interface, action budget, and evaluator together—not the base model alone.

812 long-horizon browser tasksEnd-state task successStateful multi-site browser work

WebArena-Verified is an audited release of the WebArena browser-agent benchmark. It rechecks task descriptions, reference answers, and evaluators, and replaces nondeterministic judging with deterministic checks where possible.

812 verified tasks; separate 258-task Hard subsetDeterministic end-state task successAudited stateful browser work

A benchmark that evaluates AI agents on multi-environment web challenges, testing navigation and task completion across diverse live web environments.

Web-agent tasksBrowser task completionOpen-web agent workflows

Vals AI benchmark for realistic financial analyst agent tasks across qualitative analysis, quantitative analysis, market work, comparables, precedents, earnings, disclosure, and modeling.

Financial analyst task categoriesMean score across repeated runsProfessional expert-task agent workflow

A quantitative-trading implementation benchmark that asks models to build backtesters under market-book liquidity and execution-delay constraints, then compares their outputs with a verifier.

3 quantitative-trading strategiesBacktester implementation scored by mean absolute errorMarket simulation and quantitative coding

A display-only provider-table GDPval rubric score for economically valuable work tasks.

Economically valuable work tasksRubric scoreProfessional agentic workflows

A display-only provider benchmark for finance-oriented tool-use and agent workflows.

Finance and banking tool-use tasksTask success rateProfessional finance-agent workflows

An index of agentic capability from Nous Research. It averages pass@1 scores on four suites, Hermes Bench, TerminalBench 4, TerminalBench Science and SkillsBench, with every run inside the Hermes Agent harness, and reports the mean cost per task beside the score.

Four suites: Hermes Bench (150 tasks), TerminalBench 4 without its 4 GPU tasks, TerminalBench Science and SkillsBenchMean pass@1 score across four suites, with mean cost per taskFrontier agentic

Tests agents on 150 tasks across 25 categories, including Hermes skills, research, diagrams and art, memory, tool use and safety. Each agent starts in a workspace with real files, some tasks add follow-up turns, and the grader checks the files and state left behind.

150 tasks across 25 categoriespass@1 scoreFrontier agentic

Tests whether a computer-use agent can learn a real accounts-payable job on the job, processing 100 vendor bills in a company ERP system with only the handbook, historical records, and mentor feedback a new hire would get.

100 vendor bills processed in sequence inside a simulated construction companyCumulative success rate over 100 billsLong-horizon computer use with offline and online continual learning

Tests whether a coding agent can explore a 256-room 3D puzzle world and collect its 100 hidden gems, using box-pushing puzzles, camera rotation, and long-horizon planning.

100 hidden gems across 256 rooms in one open worldGems collected out of 100, with rooms visited as a second scoreLong-horizon spatial planning, with single puzzles that can take over one hundred moves

A function-calling benchmark for tool selection, schema adherence, and argument correctness, covering single-turn, parallel, irrelevance and multi-turn subsets.

Function-calling tasksTool invocation and schema evaluationAdvanced tool use

Measures how reliably code-review agents identify useful issues in real pull requests while avoiding false positives.

219 pull requests across 187 repositories and 19 languagesGrounded precision, recall, and F1; augmented metrics

A Ramp Labs public leaderboard for long-horizon accounting agent work. BenchLM shows it as display-only reference data and excludes it from model rankings.

137 accounting tasks · 3 attempts per taskMean criterion-level score across three attemptsLong-horizon accounting agent evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

27 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

34 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

20 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

21 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

Tests agents controlling a simulated robot arm and humanoid through motor commands, sensor readings, and camera images.

24 simulated robot-control tasksMean task score out of 100Real-time perception, planning, and motor control

SciUniverse Level 1 tests whether models carry out scientific work across chemistry, biology, and materials science. Its 92 tasks include physical experiments, simulated lab work, and measurement analysis, with Pass@1 averaged equally across 17 task families.

92 tasks in 17 task familiesScientific work through a facility harness; Pass@1Level 1 scientific tasks

A 29-model comparison of how well models use Parallel Search Fast and Extract to answer web research questions.

100 questions each from DeepSearchQA, HLE, and WISEREqual-weight Search Intelligence Score (0–100)Multi-step web research and specialist questions

Tests autonomous AI research on compression, language-model training, judge-harness engineering, and post-training under fixed compute and time budgets.

4 autonomous research tasksMean reference-anchored log score, expressed as a percentageAI research under fixed compute and time budgets

A Vals AI comparison of native provider search and Exa across finance-analysis and legal-research tasks.

Finance Agent Benchmark v2 and Legal Research Benchmark tasksAccuracy by model and search-tool combinationProfessional web research with controlled search-tool variants

A Vals AI agent benchmark that gives each system five days to build and run a space program in Kerbal Space Program.

30 progressively harder Kerbal Space Program missionsMission-ladder progress with partial creditLong-horizon autonomous computer use and planning

A Vals AI computer-use benchmark in which agents play six commercial video games with a keyboard and mouse.

Six commercial video-game control tasks, including held-out gamesOverall score with game-level task splitsKeyboard-and-mouse computer use in real-time games

An archived Vals benchmark covering retrieval, numerical reasoning, financial modeling, market analysis, earnings, trends, and adjustments.

11 financial analyst task viewsAccuracy with task-level breakdownsProfessional financial analysis

A contamination-resistant agent benchmark for reverse engineering real-world binaries.

Real-world binary reverse-engineering tasksFully solved rate and capability scoreAgentic reverse engineering

A Vals-hosted view of Terminal-Bench 1.0 with easy, medium, and hard task splits.

Terminal tasks split by easy, medium, and hard difficultyAccuracy scoreTerminal-agent execution

A contamination-controlled benchmark that asks agents to reverse engineer compiled binaries without source access and satisfy six task-specific objectives per challenge.

262 binary reverse-engineering instancesFully solved challenge rate (pass@1)Binary reverse engineering

A cybersecurity benchmark for evaluating LLM agents on full-control V8 exploit synthesis using 16 measured exploit capability flags.

V8 exploit synthesis runsCapability coverage percentage over 16 flagsBrowser exploitation and cybersecurity

Number of advanced cyber-range challenges solved in the joint NIST CAISI and UK AISI preliminary evaluation.

41 advanced cyber-range challengesChallenges solvedAdvanced cyber operations

Average step reached on a 32-step long-horizon cyber range.

32-step long-horizon cyber rangeAverage step reachedLong-horizon cyber operations

Share of runs that completed the 32-step cyber range within the 100-million-token limit.

10 long-horizon runsCompletion rateLong-horizon cyber operations

Cybersecurity benchmark for agentic vulnerability analysis and exploit-oriented security tasks.

Security engineering tasksSuccess rateAdvanced cybersecurity

Independent evaluation of AI agents against vulnerable real-world systems in dynamic environments.

197 dynamic cyber tasksTasks solvedEasy through elite cyber operations

Average success rate across realistic, long-horizon cybersecurity scenarios.

11 cyber scenariosAverage success rateLong-horizon cybersecurity

Number of CyScenarioBench scenarios completed successfully in at least one run.

11 cyber scenariosScenarios solved at least onceLong-horizon cybersecurity

Irregular's domain-level evaluation of network attack simulation capability.

Atomic cyber tasksDomain averageNetwork attack simulation

Irregular's domain-level evaluation of vulnerability research and exploitation capability.

Atomic cyber tasksDomain averageVulnerability research and exploitation

Irregular's domain-level evaluation of cybersecurity evasion capability.

Atomic cyber tasksDomain averageCybersecurity evasion

Share of SCONE smart-contract vulnerabilities exploited on the 12-task post-cutoff set.

12 post-cutoff smart-contract vulnerabilitiesBest@8 exploit success rateSmart-contract exploitation

Simulated value captured across the SCONE post-cutoff smart-contract set.

12 post-cutoff smart-contract vulnerabilitiesSimulated USD millions, Best@8Smart-contract exploitation

Share of patched Firefox 147 JavaScript-engine targets for which the model produced a working arbitrary-code-execution exploit.

250 Firefox 147 vulnerability trialsWorking arbitrary-code-execution rateBrowser exploit development

Share of evaluated OSS-Fuzz entry points where the model produced at least a crash.

Approximately 830 OSS-Fuzz entry pointsAny-crash rateVulnerability discovery

Share of evaluated OSS-Fuzz entry points where the model achieved a write primitive or stronger result.

Approximately 830 OSS-Fuzz entry pointsWrite-primitive-or-higher rateVulnerability exploitation

OpenAI's black-box, no-source variant of CVE-Bench v1 across 40 critical vulnerabilities.

40 critical CVEsPass@1 over three rolloutsBlack-box vulnerability exploitation

A benchmark for agentic professional workflows with verifiable success criteria, reporting pass rates and partial scores for model plus agent-harness rows.

152 ALE-V1 professional workflow tasks across 13 top-level domainsPass rate, partial-credit score, cost, token, and duration metadataReal-world agentic workflows

An agentic coding benchmark where models use a TypeScript SDK to play a RuneScape-like environment and optimize skill-training performance.

16 RuneScape skill-training tasksAverage log XP-rate scoreAgentic gameplay automation

An independent agentic-intelligence evaluation from Toloka using private simulated workflows and a pass^5 metric.

Private simulated enterprise workflowspass^5 arena scoreAgentic workflow reliability

Evaluating agents on Excel-based financial modeling tasks

Excel-based financial modeling tasksAccuracy scoreProfessional finance modeling

Can autonomous agents craft PoC inputs that trigger OSS-Fuzz vulnerabilities—and stop crashing after the fix?

OSS-Fuzz PoC and patch-verification tasksAccuracy scoreAutonomous cybersecurity exploit reproduction

Independent agent runs on biological data investigations, split by human-solvable and human-difficult tasks.

End-to-end bioinformatics investigations from raw dataAverage task accuracy across three runsAgentic biological analysis

Agent evaluation on research-grade US tax questions using the Tax Agent Bench harness.

US tax research, calculations, forms, filings, and precedent analysisOverall score; all-pass score shown separatelyResearch-grade US tax questions

Tests an agent's ability to complete legal work using documents, spreadsheets, presentations, and file-system tools

Legal agent work across documents, spreadsheets, presentations, and filesAccuracy scoreProfessional legal workflow automation

Evaluating agents on legal research tasks across diverse areas of US law

US-law legal research tasksAccuracy scoreProfessional legal research

How important are skills for agents?

Agent skill-importance tasksAccuracy scoreAgent skill evaluation

Which model can make the most money playing poker?

Poker-playing agent trialsAccuracy scoreStrategic game-agent decision making

Can AI help people navigate SNAP benefits?

SNAP public-benefits navigation tasksAccuracy scorePublic-benefits policy navigation

Can AI help people navigate SNAP benefits?

SNAP public-benefits navigation tasksAccuracy scorePublic-benefits policy navigation

Tests scientific discovery by asking agents to infer undisclosed rules across 70 interactive text-based games.

70 interactive games across seven difficulty tiersExpected composite score with confidence intervalUnknown-rule discovery and interactive reasoning

Tests long-horizon coherence by asking agents to operate a simulated vending-machine business for one year.

One year of simulated autonomous business operation per runArithmetic mean final bank balance in USDLong-horizon agent coherence and business operation

Can agents rediscover sealed mathematical mechanisms through bounded experiments?

Sealed mathematical mechanisms probed under an experiment budgetAccuracy scoreAgentic scientific discovery

Professional-services tasks across investment banking, management consulting, and corporate law. This table shows Mercor-run configurations for reference and is excluded from model rankings.

240 tasks in the version 1.1 revisionPass@1Source-specific evaluation

Month-end accounting tasks that require work across files and applications. This table shows Mercor-run configurations for reference and is excluded from model rankings.

160 held-out accounting tasksMean ScoreSource-specific evaluation

Finding difficult information through web research. This table shows Mercor-run configurations for reference and is excluded from model rankings.

100 Mercor tasks; 130 tasks in the separately reported original setPass@1Source-specific evaluation

Producing workplace deliverables from reference files. This table shows Mercor-run configurations for reference and is excluded from model rankings.

132 Mercor tasks; 213 tasks in the separately reported original setMean ScoreSource-specific evaluation

Completing tasks in command-line environments. This table shows Mercor-run configurations for reference and is excluded from model rankings.

99 Mercor tasks; 89 tasks in the separately reported original setPass@1Source-specific evaluation

Producing workplace deliverables from reference files. This table shows Mercor-run configurations for reference and is excluded from model rankings.

213 source-reported tasksMean ScoreSource-specific evaluation

Writing research reports from web evidence. This table shows Mercor-run configurations for reference and is excluded from model rankings.

130 source-reported tasksMean ScoreSource-specific evaluation

Producing legal work across practice areas and contract tasks. This table shows Mercor-run configurations for reference and is excluded from model rankings.

1,749 source-reported tasksPass@1Source-specific evaluation

Negotiating and revising contracts across multiple turns. This table shows Mercor-run configurations for reference and is excluded from model rankings.

140 source-reported tasksTurn-Weighted Mean ScoreSource-specific evaluation

Handling simulated banking-support conversations with tools. This table shows Mercor-run configurations for reference and is excluded from model rankings.

97 source-reported tasksPass@1Source-specific evaluation

Post-training four base language models across seven weighted benchmarks, with ten hours and one H100 per run.

Post-training Qwen3 1.7B and 4B, SmolLM3 3B, and Gemma3 4BWeighted aggregate across four base models and seven benchmarksFrontier agent evaluation

Program-reconstruction hidden-test pass rate after the first of five sequential long-context episodes.

166 golden program-reconstruction tasksHidden-test pass rate after episode 1Long-context clean-room software engineering

An independently evaluated LiveCodeBench result from Artificial Analysis.

Contamination-resistant coding tasksPass rateCompetitive programming

Terminal-Bench 2.1 results, mostly provider-reported, stored separately from the Terminal-Bench 2.0 lane.

Terminal-based software-agent tasksInteractive task success rateProfessional software engineering

HumanEval tests whether generated Python functions pass unit tests on 164 handwritten problems. These scores describe function-level correctness; they do not measure repository work or a coding agent.

164 problemsPython function generationIntroductory to intermediate programming

A code-generation benchmark reported in DeepSeek-V4 base-model evaluations.

Code generation tasksPass@1Software engineering

Competitive-programming rating reported for DeepSeek-V4 thinking-mode evaluations.

Competitive programming contestsRatingElite competitive programming

A benchmark for agentic software engineering tasks executed in real terminal environments. DeepSeek reports it in the agentic section, while BenchLM also mirrors it in coding for models that publish it as a developer-task signal.

Terminal-based software tasksInteractive CLI agent evaluationProfessional software engineering

SWE-bench Verified tests patches for 500 human-filtered GitHub issues. A score measures how many tasks an evaluated model-and-agent setup resolved; the model name alone does not identify that setup.

500 verified issuesCode patch generationProfessional software engineering

A continuously updated software engineering benchmark by Nebius using fresh GitHub issues to avoid contamination. Models are evaluated 5 times per problem under a fixed ReAct scaffolding; the Resolved Rate (best pass@1) is reported.

Fresh GitHub issues (rolling window)Code patch generationProfessional software engineering

A continuously updated coding benchmark built from newly collected LeetCode, AtCoder, and Codeforces problems. Fresh problem windows reduce one contamination path, but results still need a release and setup check.

Continuously updated contest problemsCompetitive-programming evaluationCompetitive programming level

LiveCodeBench v6 is a named release slice used in provider comparison tables. Keeping it separate prevents v6 results from being mixed into older or rolling LiveCodeBench windows.

Fresh programming problemsProvider-published v6 competitive programming resultsCompetitive programming level

LiveCodeBench v5 is a named release and date-window slice. BenchLM keeps explicitly labeled v5 rows outside the rolling weighted lane.

July 2024 to May 2025 release windowProvider-published v5 competitive programming resultCompetitive programming level

This lane contains DeepSeek's LiveCodeBench Pass@1-COT results. The explicit metric and prompting label keeps them separate from generic and version-specific LiveCodeBench rows.

DeepSeek-V4 report evaluation windowPass@1-COT competitive programming resultsCompetitive programming level

A harder competitive-programming benchmark family built from Codeforces, ICPC, and IOI problems, with quarter-specific public leaderboards and difficulty-aware reporting.

Quarter-specific contest programming setsCompetitive programmingHigh-end contest programming

A repository-level Lean 4 proof engineering benchmark that measures whether a model can complete formal proofs and correctly define new mathematical concepts inside realistic FLT project pull requests.

FLT project pull requestsLean 4 repository task completionFormal verification / proof engineering

A long-horizon repository benchmark built to test realistic software engineering work. Its scores need a task-quality and setup check before they support a coding-agent decision.

1,865 repository problemsRepository task completionLong-horizon professional engineering

A Snorkel AI benchmark of senior-level software engineering tasks emphasizing under-specified feature work, bug/performance investigation, and taste-based correctness.

Senior-level repository tasksAgentic software-engineering evaluationProfessional senior engineering

An open software-engineering benchmark built from real merged post-cutoff pull requests across Python, Rust, TypeScript, JavaScript, and Go repositories.

23 post-cutoff repository tasks in the v3 reportPass@1 with low, medium, and high effortProfessional multi-file software engineering

An app-level coding benchmark that asks DevEco Code configurations to implement observable behavior in buildable OpenHarmony ArkTS applications.

153 app-development and bug-fix tasksTask completion through DevEco CodeEnd-to-end OpenHarmony application development

A post-cutoff software-engineering benchmark with hidden functional tests and regression guards, reported for vendor coding-agent harnesses.

38 validated post-cutoff repository tasksPass@1 with vendor coding-agent harnessesMid-band frontier software engineering

Cognition's 100-task software-engineering benchmark for whether coding agents produce mergeable, production-quality pull requests, scored for correctness, tests, scope, style, and maintainability through maintainer-authored rubrics.

100 private Main tasks (150 in Extended)Repository task completion with maintainer rubricsFrontier coding-agent quality

Cognition's 150-task Extended subset of the FrontierCode 1.1 software-engineering benchmark.

150 private software-engineering tasksRepository task completion with maintainer rubricsFrontier coding-agent quality

An 80-task software-engineering benchmark across eight repositories that tests whether autonomous IDE agents can explore, edit, run, and verify code changes end to end.

80 tasks across 8 repositoriesAutonomous IDE-agent task completion (pass@1)End-to-end software engineering

A six-task full-stack web-app benchmark that measures how much required functionality an AI builder or coding assistant delivers from one prompt without human code edits.

6 full-stack app-building tasksBest-of-three one-shot feature completionProduction-style full-stack application generation

A multilingual software-engineering benchmark for real-world code issue resolution across multiple programming languages.

Multilingual software-engineering tasksRepository task completionProfessional software engineering

A multimodal variant of SWE-bench that adds visual context such as screenshots and design mockups to software engineering issue descriptions.

Multimodal software engineering tasksCode patch generation with visual contextFrontier multimodal coding

Cursor's first-party benchmark for ambiguous, multi-file coding-agent tasks drawn from real Cursor sessions, currently on the 4.0 task set.

Long-horizon multi-file agentic coding tasksCursor agent-loop evaluationProfessional agentic software engineering

A multi-language software-engineering benchmark that measures repository-level bug fixing and implementation across more than one programming ecosystem.

Multi-language repo tasksRepository task completionProfessional software engineering

A repo-level code generation and full-project delivery benchmark spanning web, mobile, and simulation-style implementation tasks.

Full project delivery tasksRepository-level implementation benchmarkEnd-to-end software delivery

Vals.ai benchmark for evaluating whether models can build complete web applications from natural language specifications in a production-like development environment.

End-to-end web application buildsFull-stack app implementation benchmarkEnd-to-end software delivery

A cleanroom software-engineering benchmark where agents receive only a compiled executable and documentation, then must architect and implement a complete codebase that reproduces the original program's behavior.

200 program reconstruction tasksCleanroom executable reimplementationFull-repository software architecture

A software-engineering benchmark for post-training infrastructure and implementation tasks, evaluated through the official Harbor implementation.

Post-training software-engineering tasksHarbor agent evaluationFrontier software engineering

An ultra-long-horizon software-engineering benchmark with open-ended implementation, performance, and research tasks designed to challenge frontier coding agents.

17 ultra-long-horizon engineering and research tasksMean@5, best@5, average rank, and dominanceUltra-long-horizon frontier software engineering

A 34-task expansion of FrontierSWE for ultra-long-horizon engineering and research work that remains far from saturation.

34 ultra-long-horizon engineering and research tasksFive-trial mean task score (Mean@5), 0-100Ultra-long-horizon frontier software engineering

A code-comprehension benchmark covering production repositories across Go, Python, C, and TypeScript.

124 codebase questions across 11 repositoriesMean pass@1Production codebase comprehension

A blind-graded coding-agent benchmark with 105 planted bugs across two production TypeScript repositories.

105 planted bugs across two production TypeScript repositoriesStrict planted bugs fixedBlind production-repository bug finding and repair

A Moonshot AI internal coding-agent benchmark for realistic software-engineering tasks across mainstream programming languages and production technology stacks.

Realistic coding-agent tasksCoding-agent pass rateProduction software engineering

A 30-task subset of MLS-Bench that evaluates whether AI systems can invent generalizable and scalable machine-learning methods.

30 machine-learning research tasksAgentic ML task evaluationML research and systems engineering

A research-reproduction benchmark that asks agents to recreate the contributions of AI papers from the paper alone.

AI research-paper reproductionLong-horizon agent evaluationFrontier autonomous research and engineering

Qwen's internal benchmark for building and rendering bilingual React projects across seven categories.

Bilingual React project constructionBradley-Terry/Elo ratingProduction frontend development

A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.

Natural language to repository tasksRepository understanding benchmarkSystem-level software comprehension

DeepSeek's internal full-stack coding-agent benchmark.

Internal full-stack coding-agent tasksProvider-reported scoreFull-stack software engineering

DeepSeek's internal hard coding-agent benchmark.

Internal hard coding-agent tasksProvider-reported scoreAdvanced coding-agent challenges

An open benchmark for AI coding agents on real-world React Native implementation tasks, emphasizing working app behavior, recommended architecture choices, and strict constraint adherence.

React Native app implementation tasksFramework-specific app development evaluationProduction mobile app engineering

A coding-agent benchmark for realistic React work, with rubrics that check production concerns such as performance, accessibility, correctness, and code quality.

51 production React tasksPass@1 weighted rubric scoreProduction frontend engineering

An agentic GPU-kernel benchmark that measures how much of the hardware roofline a model's correct, audit-clean kernels reach on six demanding CUDA and Triton problems.

6 GPU-kernel optimization problemsMean peak fraction of hardware roofline over valid cellsAgentic GPU systems engineering

A Vercel benchmark for AI coding agents on Next.js code generation and migration tasks, reporting pass@4, average execution time, and results with bundled documentation through AGENTS.md.

31 current Next.js code generation and migration tasksSuccess rate (pass@4), with and without bundled documentationFramework-specific web application engineering

A text-to-SQL benchmark over realistic warehouse-scale schemas, reported by Interfaze for model comparison.

Text-to-SQL queriesExecution accuracyEnterprise text-to-SQL

SciCode evaluates language models on generating code for realistic scientific research problems across 16 subfields of physics, math, chemistry, biology, and material science. Problems decompose into 338 subproblems requiring domain knowledge recall, scientific reasoning, and precise code synthesis. Based on real scripts from published research.

80

A display-only Artificial Analysis coding index.

Cross-benchmark coding indexAggregated model scoreDisplay-only external reference

A display-only Artificial Analysis leaderboard for coding-agent systems, combining agent harnesses, host models, and execution settings across software-engineering benchmarks.

Composite over DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnAAverage pass@1 indexReal-world coding-agent workflows

An Artificial Analysis SciCode score.

Scientific coding subproblemsTask success rateScientific programming

A display-only Artificial Analysis coding metric for agentic coding and terminal use on a harder Terminal-Bench slice.

Agentic coding and terminal tasksTask success rateProfessional software engineering

A display-only MiniMax provider benchmark for end-to-end coding-agent and product-building tasks.

End-to-end coding-agent tasksTask success rateFrontier coding-agent workflows

A display-only provider benchmark for generating or manipulating SVG outputs from natural-language requirements.

SVG generation and editing tasksTask success rateVisual coding and structured graphics generation

A display-only benchmark for difficult GPU kernel implementation and optimization tasks.

Hard GPU kernel coding tasksTask success rateSpecialized systems programming

Evaluates coding agents on 333 multimodal game-development tasks in Godot, spanning 2D graphics, 3D graphics, user interfaces, and gameplay logic.

333 tasks from 88 tutorialsPass@1 on the full task set with 95% confidence intervalsMultimodal game development in Godot 4.4.1

A systems and software-engineering benchmark from ByteDance Seed that evaluates agents on long-horizon edge tasks using time-budgeted learning curves rather than a single static pass rate.

Systems and software-engineering tasksTime-budgeted agent learning curvesLong-horizon engineering

Vals AI’s independent implementation of the LiveCodeBench code-generation benchmark, run under one fixed harness across the models it tracks.

Competitive programming problems (easy, medium, hard)Pass@1 accuracyFrontier coding

Vals AI’s independent run of the public SWE-bench issue set, reported by human time-to-fix bucket. Vals removed SWE-bench Verified from its index as saturated on 2026-05-04; this board is the standalone SWE-bench run.

Real repository issues by human time bucketResolved rateFrontier coding agents

An electrical-engineering agent benchmark maintained by atopile.

Electrical-engineering design tasksAgent task scoreProfessional electrical engineering

Hugging Face's open benchmark for generating valid and geometrically correct 3D mechanical parts.

49 CAD generation samplesValidity-gated CAD scoreProfessional mechanical CAD

A long-horizon software engineering benchmark from Datacurve for measuring frontier coding agents on original tasks drawn from active open-source repositories.

113 software engineering tasks across 91 repositories and 5 languagesPass@1 with confidence interval, cost, time, and token metadataLong-horizon software engineering

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

15 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

15 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

24 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

17 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

24 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

JavaScript issue resolution where bug reports depend on screenshots, evaluated by Vals AI with no agent internet access.

480 JavaScript issues with visual bug reportsResolved issue percentageVisual software issue resolution

BridgeBench rates 25 models across nine software-building axes, including reasoning, front end, back end, security, speed, and cost. Its overall rating is the equal-weight mean of those axes.

No public task countEditorial Elo-style ratingSoftware-building assessments

A long-horizon software engineering benchmark from Abundant AI with multi-hour tasks spanning library reproductions, full-stack product clones, and ML engineering.

20 multi-hour software engineering tasksTask resolution and trajectory reviewUltra-long-horizon software engineering

An agentic coding benchmark that asks models to build a Game Boy Advance emulator from scratch and grades emulator behavior against procedural, audio, and gameplay tests.

27 emulator test casesOverall emulator scoreLong-horizon systems programming

A machine-learning engineering benchmark that tests whether LLMs can train models on novel datasets, write PyTorch code, and improve through iterative feedback.

17 novel ML engineering tasksAverage accuracy across tasksNovel dataset modeling and iterative debugging

Can language models reimplement real-world programs in another language?

Real-world program reimplementation in another languageAccuracy scoreProduction code migration

Based on the International Olympiad in Informatics

International Olympiad in Informatics-style programming tasksAccuracy scoreOlympiad programming

Archived Vals AI results on 2024 and 2025 International Olympiad in Informatics programming tasks.

IOI 2024 and 2025 programming tasksAccuracy scoreOlympiad programming

Can language models rebuild programs from scratch?

Program reconstruction tasksAccuracy scoreCleanroom software engineering

Can a model extend one working web application across a long sequence of dependent requests?

Long chains of dependent web-application change requestsAccuracy scoreLong-horizon software delivery

Software integration and production-failure diagnosis with the Terminus-2 harness. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Integration and observability engineering tasksPass@1Source-specific evaluation

Software integration and production-failure diagnosis with the Inspect harness. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Integration and observability engineering tasksPass@1Source-specific evaluation

Answering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings.

50 Mercor tasks; 124 tasks in the separately reported original setPass@1Source-specific evaluation

Editing repositories to resolve verified GitHub issues. This table shows Mercor-run configurations for reference and is excluded from model rankings.

50 Mercor tasks; 500 tasks in the separately reported original setPass@1Source-specific evaluation

Answering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings.

124 source-reported tasksPass@1Source-specific evaluation

Solving scientific programming subproblems with executable checks. This table shows Mercor-run configurations for reference and is excluded from model rankings.

65 source-reported tasksMean ScoreSource-specific evaluation

Hardware design, verification, and comprehension tasks. This table shows Mercor-run configurations for reference and is excluded from model rankings.

517 source-reported tasksPass@1Source-specific evaluation

Resolving software issues on the DeepSWE v1.1 task set. This table shows Mercor-run configurations for reference and is excluded from model rankings.

113 source-reported tasksPass@1Source-specific evaluation

Reconstructing a program from its executable behavior and documentation. This table shows Mercor-run configurations for reference and is excluded from model rankings.

100 source-reported tasksPass@1Source-specific evaluation

Coding agents recreate nine reference web apps from an empty directory. Plato checks visual and functional behavior through end-to-end workflows.

9 reference web appsPublished full score (%)Long-running web application recreation

Scores how far a model advances 120 open research tasks across biology, chemistry, physics, astronomy, Earth science and engineering on a 0 to 4 progress scale, crediting supported partial work and honest statements of what remains unresolved.

120 research tasks, 20 in each of six STEM domainsAudited progress level 0 to 4 per task, refined to a fractional score; complete-run mean across all tasksOpen research targets; no run has reached a full-target success

Breadth-first-search graph traversal on 200 problems with context lengths from 256K to 1M tokens.

Breadth-first-search graph traversal on 200 problems with context lengths from 256K to 1M tokens.F1Frontier agent evaluation

ARC Prize fluid-intelligence benchmark using novel visual grid transformations.

Semi-private ARC-AGI-1 evaluation setVerified accuracyAbstract visual reasoning

A dataset for evaluating language models on multistep soft reasoning tasks specified in natural language narratives. Tests the ability to perform complex, structured reasoning.

Multi-step reasoningNarrative-based reasoningComplex reasoning tasks

A suite of 23 challenging tasks from the BIG-Bench collaborative benchmark where prior language models failed to exceed average human performance, even with chain-of-thought prompting.

23 tasksMixed reasoning tasksAdvanced reasoning

A reading-comprehension benchmark requiring discrete reasoning over paragraphs, reported in DeepSeek-V4 base-model evaluations.

Paragraph reasoning questionsF1Reading and numerical reasoning

A commonsense natural-language inference benchmark reported in DeepSeek-V4 base-model evaluations.

Commonsense completion questionsExact matchCommonsense reasoning

A commonsense coreference benchmark reported in DeepSeek-V4 base-model evaluations.

Coreference resolution questionsExact matchCommonsense reasoning

A Chinese Winograd Schema Challenge benchmark reported in DeepSeek-V4 base-model evaluations.

Chinese coreference questionsExact matchChinese commonsense reasoning

A word-chain reasoning benchmark that tests planning, recall, constraint following, and vocabulary depth by asking models to extend non-repeating edit-distance-1 chains.

50 starting words × 3 trialsDifficulty-weighted word-chain reasoningOpen-ended lexical planning

Tests whether model judgments rank argumentative critiques in the same order as expert human ratings across philosophy, AI alignment, and other concept-heavy texts.

224 texts and 608 within-text critique pairsAverage pairwise-ranking loss against expert ratingsFuzzy, expert-rated argumentative reasoning

A multi-step verifiable reasoning benchmark that evaluates whether models can solve pencil puzzles with unique solutions.

300 evaluation puzzlesDirect and agentic puzzle solve rateMulti-step verifiable reasoning

A long-context benchmark that measures whether models can actually use extended context windows for reasoning and retrieval.

Long-context tasksExtended-context retrieval and reasoningHard long-context

A long-context benchmark for memory, retrieval, and multi-round coherence over large contexts.

Long-context retrievalMulti-round long-context evaluationHard long-context

MRCR v2 slice focused on long-context retrieval at 64K-128K lengths.

8-needle retrieval tasksLong-context retrievalLong-context reasoning

MRCR v2 slice focused on very long contexts at 128K-256K lengths.

8-needle retrieval tasksVery-long-context retrievalVery long-context reasoning

MRCR v2 slice focused on retrieval across 256K-512K-token contexts.

100 eight-needle retrieval examplesMean sequence-matcher ratioVery long-context retrieval

MRCR v2 slice focused on retrieval across 512K-1M-token contexts.

100 eight-needle retrieval examplesMean sequence-matcher ratioMillion-token retrieval

Long-context graph traversal benchmark using breadth-first search tasks.

Graph traversal tasksLong-context graph reasoningAlgorithmic long-context reasoning

Long-context benchmark for recovering parent relationships inside graph tasks.

Graph parent-retrieval tasksLong-context graph reasoningAlgorithmic long-context reasoning

A million-token MRCR long-context retrieval benchmark reported in DeepSeek-V4 model evaluations.

Million-token retrievalLong-context retrieval MMRMillion-token long context

A million-token CorpusQA long-context question-answering benchmark reported in DeepSeek-V4 model evaluations.

Million-token corpus question answeringLong-context QA accuracyMillion-token long context

ARC-AGI-2 tests whether a system can infer a transformation from example grids and apply it to a new grid. Compare accuracy with the evaluation split, attempt limit, and cost attached.

Visual pattern completion and abstract reasoningGrid transformation puzzles with novel rulesExpert-level — hardest public reasoning benchmark

An interactive successor to ARC-AGI-2 that evaluates whether an AI agent can learn unfamiliar task mechanics through action and feedback.

Interactive game-like tasks with hidden rulesAgentic task completion under a capped evaluation budgetFrontier agentic reasoning

A multistage statistical-reasoning benchmark for genomics and biological-data analysis agents.

129 genomics statistical-analysis workflowsEval-level pass rate across dependent analysis decisionsLong-horizon scientific reasoning

A long-context retrieval benchmark that measures whether a model can recover relevant information embedded deep inside very long contexts.

Long-context retrievalNeedle-in-a-haystack recallLong-context memory

The hardest subset of GPQA featuring the most challenging graduate-level science questions. Sometimes reported separately from the standard GPQA benchmark.

Expert-level science questionsMultiple choice questionsGraduate-level scientific reasoning

An Artificial Analysis long-context reasoning evaluation.

Long-context reasoning tasksAccuracyLong-context reasoning

An open benchmark from Wisedocs, run by Artificial Analysis, measuring how well models reason over long, fragmented medical records with multi-document synthesis.

Long, fragmented medical-record reasoningAccuracyLong-context medical reasoning

An Artificial Analysis metric for research-level physics reasoning.

Research-level physics questionsAccuracyResearch-level physics reasoning

A benchmark that tests whether AI models challenge nonsensical, ill-posed, or logically flawed prompts instead of confidently generating incorrect answers. Measures the critical ability to push back on bad input.

Nonsensical and flawed prompts across multiple domainsPrompt challenge and refusal evaluationRobustness and critical reasoning

An automated evaluation framework using 1,000+ real-world user tasks covering reasoning, planning, coding, and creative writing. Highly correlated with Chatbot Arena human preference rankings.

1,024 real-world tasksReal-world task evaluationDiverse real-world scenarios

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

5 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

14 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

Selected original RULER context sweeps and separate IBM Granite 4.2 reports. The table keeps each source cohort and context length explicit; 128K is the displayed score, and the results do not enter rankings.

13 tasks in the original author cohort; IBM task count undisclosedReported score by context length

Selected author accuracy curves for long-context retrieval with minimal wording overlap. Original NoLiMa stays separate from Hard, and 64K/128K use fewer placements. The displayed score is 32K accuracy, outside overall and category rankings.

58 question-needle pairs; five haystacks; 26 placements through 32KCharacter-name answer accuracy by context length

Selected reasoning-model accuracy curves on the ten hardest NoLiMa question-needle pairs. Exact API snapshots and effort remain undisclosed. The displayed score is 32K accuracy; these reports stay outside overall and category rankings.

10 hardest question-needle pairsHard-subset character-name answer accuracy

Synthesizing information across long collections of documents. This table shows Mercor-run configurations for reference and is excluded from model rankings.

89 Mercor tasks; 100 tasks in the separately reported original setPass@1Source-specific evaluation

Synthesizing information across long collections of documents. This table shows Mercor-run configurations for reference and is excluded from model rankings.

100 source-reported tasksPass@1Source-specific evaluation

Professional chart understanding across 100 specialized chart types with expert-set answer tolerances.

100 specialized chart typesAccuracy without toolsProfessional chart reasoning

Professional chart understanding with a container, standard libraries, and image cropping.

100 specialized chart typesAccuracy with toolsProfessional chart reasoning

Generates CadQuery code from multi-view renders and scores geometric similarity by voxel intersection-over-union.

1,000-file Vision2Code subsetVoxel IoUProgrammatic CAD generation

Generates CadQuery code from multi-view renders with image inspection and code-execution tools.

1,000-file Vision2Code subsetVoxel IoU with toolsProgrammatic CAD generation

Quantitative biomedical image analysis across segmentation, detection, measurement, classification, registration, reconstruction, and tracking tasks.

91 cases across 31 biomedical imaging tasksNormalized task metricBiomedical image analysis

Professional document understanding over 100 real-world PDFs from ten domains.

100 professional document promptsMean criteria pass rateProfessional document reasoning

Professional document understanding with a container, standard libraries, and image cropping.

100 professional document promptsMean criteria pass rate with toolsProfessional document reasoning

Grounded numerical reasoning over a corpus of historical U.S. Treasury Bulletin documents.

Historical Treasury Bulletin questionsAgentic grounded QA accuracyProfessional document reasoning

A broad multimodal reasoning benchmark spanning charts, diagrams, tables, and academic visual question answering.

Multimodal academic reasoningImage + text question answeringFrontier multimodal

A harder multimodal benchmark for frontier models that combines text with images, diagrams, charts, and academic visual reasoning tasks.

Multimodal academic reasoningImage + text question answeringFrontier multimodal

A native OCR benchmark for reading text from images across multilingual scripts, low-quality scans, handwriting, structured layouts, charts, and screenshots.

Image OCR tasksAccuracyNative visual text understanding

An end-to-end document understanding benchmark over long, layout-rich PDFs with tables, equations, headers, footnotes, and multi-column flows.

Layout-rich PDF understandingMean accuracyComplex document processing

A speech-recognition benchmark on the cleaned Artificial Analysis VoxPopuli subset, reported as word error rate where lower is better.

Speech-to-text transcriptionWord error rateAudio speech recognition

A display-only Design Arena website-generation Elo score surfaced on OpenRouter model benchmark pages.

Website generation comparisonsEloDesign and website generation

A benchmark for grounded reasoning over office-style documents, spreadsheets, charts, and business artifacts.

Document and spreadsheet tasksGrounded QA over office artifactsEnterprise grounded reasoning

A tool-augmented MathVision variant that permits Python during visual mathematics reasoning.

Visual mathematics problems with PythonImage and mathematics reasoning with toolsAdvanced multimodal mathematics

A Python-assisted BabyVision evaluation for fine-grained visual perception and grounded reasoning.

Visual perception tasks with PythonTool-augmented multimodal scoreFine-grained visual perception

A Python-assisted ZeroBench_main evaluation reported as pass@5.

Visual reasoning questions with PythonPass@5Tool-augmented visual reasoning

A forced-answer WorldVQA variant for atomic visual world knowledge.

Atomic visual world-knowledge questionsForced-answer visual QAFine-grained visual knowledge

A document-understanding benchmark for parsing and reasoning over complex document layouts.

Complex document-understanding tasksDocument-understanding scoreGrounded document reasoning

Moonshot AI's internal benchmark for atomic visual perception capabilities.

Internal atomic visual-perception tasksInternal evaluation scoreFine-grained visual perception

Tool-augmented MMMU-Pro variant that allows Python assistance during multimodal reasoning.

Multimodal academic reasoningImage + text question answering with PythonFrontier multimodal

A document understanding benchmark used in frontier-model comparison tables to measure extraction and grounded reasoning quality on complex documents.

Document understanding tasksDocument understanding benchmarkGrounded document reasoning

A display-only Liquid AI extraction metric measuring the share of image-to-JSON outputs that parse as strict JSON.

Image-to-JSON extractionStrict JSON parseability rateStructured visual extraction

A display-only Liquid AI extraction metric measuring field-name agreement between requested schema fields and extracted JSON fields.

Image-to-JSON extractionSchema field F1Structured visual extraction

A display-only Liquid AI extraction metric measuring judged agreement between extracted values and the source image.

Image-to-JSON extractionVLM-judged extraction accuracyStructured visual extraction

A grounded visual QA benchmark focused on answering practical questions about real-world images and scenes.

Real-world visual question answeringImage-grounded QAGeneral visual reasoning

A video understanding benchmark that allows subtitle access when answering multimodal questions about videos.

Video understandingVideo QA with subtitle contextMultimodal video reasoning

A stricter Video-MME setting that removes subtitle help and tests video understanding from visual and audio context alone.

Video understandingVideo QA without subtitle contextMultimodal video reasoning

A comprehensive benchmark for multimodal large language models on video understanding, covering temporal reasoning, perception, and question answering over videos.

Video understandingVideo QA and analysisBroad multimodal video reasoning

A visual mathematics benchmark that tests whether a model can solve math problems grounded in diagrams, equations, figures, and other visual inputs.

Visually grounded math problemsImage + math reasoningAdvanced multimodal mathematics

A multimodal math benchmark for visually grounded mathematical reasoning and answer generation.

Visually grounded math problemsMultimodal mathematical reasoningAdvanced multimodal mathematics

A multimodal benchmark for dynamic mathematical reasoning over visual and structured inputs.

Dynamic visual math problemsMultimodal mathematical reasoningAdvanced multimodal mathematics

A general visual question-answering benchmark used in provider tables for real-image reasoning quality.

Real-image visual QAImage-grounded QAGeneral visual reasoning

A conversational visual QA benchmark that tests multi-turn grounded answering over images and documents.

Conversational visual QAMulti-turn image-grounded QAConversational multimodal reasoning

A long-document multimodal benchmark for grounded reasoning over extended document contexts.

Long document understandingDocument-grounded reasoningLong-context document reasoning

An OCR-focused benchmark for reading and extracting text from visually complex documents and images.

Optical character recognitionText extraction from images and documentsDocument reading

A diagram understanding benchmark focused on scientific and educational visual question answering.

Diagram understandingDiagram-grounded QAStructured visual reasoning

A visual counting benchmark that tests whether a model can count objects and entities reliably in complex scenes.

Visual counting tasksImage-grounded countingFine-grained visual perception

A referring-expression grounding benchmark averaged across RefCOCO variants to test whether a model can localize described objects correctly.

Referring-expression groundingGrounded visual localizationFine-grained visual grounding

A visual detection and grounding benchmark slice used to compare zero-shot object understanding across diverse domains.

Out-of-distribution object understandingDetection and groundingRobust visual grounding

A grounded visual reasoning benchmark focused on evidence-based question answering over real images.

Evidence-based visual QAGrounded image reasoningGrounded multimodal reasoning

A video extension of MMMU-style multimodal reasoning over expert questions grounded in temporal media.

Video-grounded expert reasoningVideo + text reasoningFrontier multimodal video reasoning

A multi-task video understanding benchmark averaged across MLVU categories.

General video understandingVideo QA and understandingBroad multimodal video reasoning

A long-video understanding benchmark for retrieving and reasoning over information distributed across extended video inputs.

Long-form video question answeringLong-video understanding scoreExtended temporal reasoning

A benchmark for evaluating multimodal models on video understanding tasks across multiple disciplines, emphasizing temporal reasoning and comprehension over video content.

Video understandingVideo reasoning benchmarkMulti-disciplinary multimodal video reasoning

A GUI-grounding benchmark for 1,581 instructions in full-screen, high-resolution professional interfaces. It tests where a target is, not whether an agent can finish the surrounding workflow.

1,581 grounding instructionsStatic interface element localizationProfessional GUI grounding

A visual agent benchmark for interface reasoning and task execution over screenshots or software surfaces.

Visual agent and interface reasoningScreenshot-grounded task reasoningComputer-use visual reasoning

An evaluation focused on professional domain expertise and task delivery quality in office-style knowledge work.

Professional office deliveryELO-style office benchmarkProfessional knowledge work

A multimodal medical multiple-choice benchmark covering clinical images such as X-rays, histology, and dermatology.

2,000 multimodal medical questionsMedical visual MCQClinical multimodal reasoning

A multi-step visual reasoning benchmark with pass@5 reporting and optional tool use.

100 visual reasoning questionsMulti-step visual reasoningTool-augmented visual reasoning

A multimodal coding benchmark for turning visual designs into working frontend implementations.

Design-to-code tasksVisual input to frontend implementationMultimodal coding

A vision-language coding benchmark for generating correct code from visual and multimodal inputs.

Multimodal coding tasksVision-language code generationMultimodal coding

A benchmark for converting visual references into functional web implementations.

Screenshot-to-web tasksVisual reference to web implementationMultimodal web generation

A multimodal retrieval and extraction benchmark over image-heavy task settings.

Visual retrieval tasksImage-grounded retrieval and extractionMultimodal retrieval

A multimodal search benchmark for retrieval and grounded answering across mixed-media inputs.

Multimodal search tasksMixed-media retrieval and grounded answeringMultimodal search

A harder MMSearch variant for multimodal retrieval and grounded tool-use workflows.

Hard multimodal search tasksAdvanced mixed-media retrieval benchmarkAdvanced multimodal search

A visual question answering benchmark focused on straightforward image-grounded understanding.

Visual QA tasksImage-grounded question answeringGeneral visual understanding

A grounded multimodal factuality benchmark for evidence-linked answer correctness.

Grounded factuality tasksEvidence-linked multimodal factualityGrounded multimodal factuality

A vision-centric benchmark for high-level multimodal reasoning and perception quality.

Frontier multimodal reasoning tasksVision-centric reasoning benchmarkFrontier multimodal

A scientific chart reasoning benchmark that tests whether models can understand, interpret, and reason about complex scientific visualizations including plots, diagrams, and data charts.

Scientific chart reasoningChart understanding and reasoningScientific visualization reasoning

Tool-free variant of CharXiv that isolates raw visual reasoning ability without code execution or tool augmentation.

Scientific chart reasoning (tool-free)Chart understanding without toolsScientific visualization reasoning

A multimodal benchmark for fine-grained visual perception and grounded reasoning tasks.

Visual perception tasksMultimodal visual reasoningFine-grained visual perception

A multimodal variant of SWE-bench that adds visual context (screenshots, design mockups) to software engineering issue descriptions, testing whether models can leverage visual information for code generation.

Multimodal software engineering tasksCode patch generation with visual contextFrontier multimodal coding

An agentic spatial reasoning benchmark reported as a normalized score.

Spatial reasoning from blueprintsNormalized scoreAgentic spatial reasoning

A scientific chart understanding benchmark scored across both the descriptive and reasoning splits rather than the reasoning split alone.

Scientific chart description and reasoningChart understanding and reasoningScientific visualization reasoning

A visual question answering benchmark whose questions cannot be answered from the image alone and require commonsense or world knowledge.

Knowledge-grounded visual question answeringMultiple choice and direct answerCommonsense and world knowledge about images

A document parsing benchmark whose overall score composites edit distance, CDM and TEDS sub-scores across diverse PDF layouts.

Document parsing and extractionComposite of edit distance, CDM and TEDSGrounded document reasoning

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

35 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

15 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

18 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

HSS tests what a model can infer from images and silent videos across temporal, spatial, social, and abstract reasoning. We show the published results as display-only evidence, excluded from overall and category rankings.

522 tasks (288 image, 234 video)Pass@1 (%) averaged over three attempts per taskIntuitive visual reasoning

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

63 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

Perplexity reports public ViDoRe V3 image retrieval nDCG@10 of 62.3 for the 0.6B model and 65.2 for the 9B model.

Public ViDoRe V3 retrievalVisual document retrieval

Perplexity reports public ViDoRe V3 Markdown retrieval nDCG@10 of 61.2 for the 0.6B model and 64.7 for the 9B model.

Public ViDoRe V3 retrievalVisual document retrieval

Vals AI multimodal composite across finance, coding, education, and mortgage-tax task families.

Finance, coding, education, and mortgage-tax componentsComposite scorePrivate multimodal economic-work benchmark composite

Vals AI benchmark for mortgage and tax document reasoning, including semantic and numerical extraction task views.

Mortgage and tax extraction tasksAccuracy scoreProfessional mortgage-tax document reasoning

A live human-preference benchmark where language models turn text prompts into voxel structures and voters compare anonymous builds from the same prompt.

Live text prompts for 3D voxel constructionGlicko-2 rating from blind pairwise votes3D spatial construction and visual quality

A live human-preference benchmark where multimodal models build voxel structures from image references and voters compare anonymous results produced from the same prompt.

Live image-reference prompts for 3D voxel constructionGlicko-2 rating from blind pairwise votesVisual grounding, 3D construction, and aesthetic quality

Tests whether speech-to-text models preserve exact structured values in English workplace audio.

300 workplace speech clips with exact structured valuesTask success rateStructured-value speech transcription

Student Assessment with Generative Evaluation

Student assessment with generative evaluationAccuracy scoreEducation assessment reasoning

MathVista tests mathematical reasoning about images. This table shows published accuracy on the testmini split, with each source configuration kept separate. The results are display only and do not enter overall or category rankings.

1,000 examples in the testmini splitPublished accuracy (%)Mathematical reasoning in visual contexts

MathVista tests mathematical reasoning about images. This table shows published accuracy on the test split, with each source configuration kept separate. The results are display only and do not enter overall or category rankings.

5,141 examples in the test splitPublished accuracy (%)Mathematical reasoning in visual contexts

DocVQA asks questions about a single document image. This page shows four historical Anthropic reports on the test split, with ANLS, zero-shot setup, report dates, and source cohorts kept visible.

Questions about single document images; provider run counts undisclosedText answers drawn from document images

ChartQA tests visual and logical reasoning about charts. These own-provider reports keep Anthropic's relaxed-accuracy results separate from Zyphra scores whose metric and scorer are not disclosed.

Provider-reported aggregates; separate source cohorts

InfographicVQA asks questions about infographic images. These two own-provider InfoVQA test reports name the split but do not disclose the metric or scorer; their printed scores remain separate source facts.

Provider-reported scores; metric/scorer not disclosed

TextVQA requires reading and reasoning about text in images. These two own-provider reports use the validation split, with metric and scorer left undisclosed in the original model cards.

Provider-reported validation scores; metric/scorer not disclosed

Answering descriptive and reasoning questions about research-paper charts. This table shows Mercor-run configurations for reference and is excluded from model rankings.

150 Mercor tasks; 5000 tasks in the separately reported original setPass@1Source-specific evaluation

Medical multiple-choice reasoning over images and clinical details. This table shows Mercor-run configurations for reference and is excluded from model rankings.

51 Mercor tasks; 200 tasks in the separately reported original setPass@1Source-specific evaluation