AI Benchmarks Directory

Anthropic's unguided vulnerability-discovery and exploitation evaluation: from a single function entry point in an open-source project, the model must find a bug and push it toward a working exploit. This lane counts control-flow hijacks.

Open-source function entry pointsCount of control-flow hijacksExploit development

Anthropic's internal evaluation of end-to-end financial-analyst work: 123 tasks, each model's attempts compared pairwise by a grader and rated on an Elo scale.

123 finance tasksElo (pairwise grader)Professional finance work

Anthropic's internal evaluation of whether a model can independently rediscover known vulnerabilities from 1,829 CVE advisories, given only the affected source code.

1,829 CVE advisoriesShare of advisories rediscoveredVulnerability discovery

The rubric pass rate on Artificial Analysis's AA-Briefcase v1.1 knowledge-work benchmark, alongside its Elo headline.

Professional knowledge-work briefsRubric pass rateProfessional knowledge work

Scale AI and the Center for AI Safety's benchmark of 240 real paid freelance projects; the automation rate is the share of projects whose deliverable a client would accept.

240 freelance projectsAutomation rate (%)End-to-end remote work

A simulated robotics benchmark that measures the average success rate of a model controlling robots across simulated manipulation tasks.

Simulated robot-control tasks (count not published)Average success rateEmbodied agent control

Perplexity's wide-search research benchmark: 500 tasks in which a model must find tens to hundreds of entities, collect facts about each, and return them as cited structured records.

500 wide-search research tasksSoft F1 (relaxed score)Agentic search

Personal-assistant agent tasks run end to end inside the OpenClaw agent harness.

Assistant agent tasksTask scoreMulti-step assistant work in a real harness

ServiceNow benchmark of stateful planning and tool use across simulated enterprise systems, as run by model providers.

Enterprise workflow tasksTask success rateStateful enterprise tool use

NVIDIA multi-turn version of When2Call that tests whether a model calls a tool, asks a follow-up question, or abstains.

Multi-turn tool-use decisions (not yet published)Overall accuracyWhen to call, clarify or abstain

Apple benchmark of stateful, conversational tool use, where tool calls change state and a simulated user takes part.

Stateful tool-use conversationsAverage accuracyTool use with changing state

Tool-hallucination benchmark that checks whether a model selects and calls tools that fit the request.

Tool-use requestsPass rateTool grounding and hallucination

Evaluates research agents against a fixed document corpus, separately from live-web BrowseComp.

Tests whether a system can discover a programming language's hidden rules through experiments and apply them to unseen tasks.

70 held-out program-synthesis tasks; 31 discovery targetsHeld-out accuracy (Best@3)Discovering and applying unfamiliar executable rules

Tests whether a system can discover altered logic rules and use them to prove or correctly decline unseen theorems.

70 held-out theorems; 24 altered rulesHeld-out accuracy (Best@3)Learning and applying unfamiliar inference rules

Google-run scientific workflow evaluation using a sixfold verifier timeout to address verification timeouts.

70 expert-curated scientific research workflowsResolution rate across 3 trials per taskFrontier scientific research workflows

Defensive vulnerability patching on 250 audit-and-patch tasks spanning 185 weakness types, 100 public and 150 held out.

Audit and patch C/C++, Go, Java, TypeScript/JavaScript, Python, and Rust repositories; 100 public and 150 held-out tasksPass@1Frontier agent evaluation

Defensive vulnerability patching on 120 held-out tasks spanning 73 weakness types. Replaced by v2 on November 2, 2026.

Audit and patch held-out C/C++, Go, Java, TypeScript/JavaScript, Python, and Rust repositoriesPass@1Frontier agent evaluation

Agentic data-analysis tasks scored against per-task rubrics at a 980K-token budget.

Agentic data research and analysis tasksNormalized rubric scoreProfessional data analysis

Average coverage of required claims in answers produced during real-world MCP tool-use workflows.

Production-like multi-server MCP workflowsMean claim coverageReal-world tool use

Strict task success requiring every expert-written legal-work rubric criterion to pass.

1,235 legal-agent tasksAll-criteria pass rateProfessional legal work

Mean fraction of expert-written rubric criteria passed across legal-agent tasks.

1,235 legal-agent tasksMean criterion-pass rateProfessional legal work

Harvey AI's strict held-out task success rate requiring every rubric criterion to pass.

Harvey-held-out legal-agent tasksAll-criteria pass rateProfessional legal work

Harvey AI's mean criterion-level score on its held-out legal-agent evaluation.

Harvey-held-out legal-agent tasksMean criterion-pass rateProfessional legal work

Fraction of Toolathlon Verified tasks solved in at least one of three trials.

108 verified real-world tool-use tasksPass@3Long-horizon application tool use

Fraction of Toolathlon Verified tasks solved in all three independent trials.

108 verified real-world tool-use tasksAll-three-trials pass rateLong-horizon application tool use

Average assistant turns per Toolathlon Verified trajectory.

108 verified real-world tool-use tasksAverage trajectory lengthLong-horizon application tool use

A continuously maintained benchmark for difficult computer work, including coding, deep learning, finance, engineering, math, and science tasks.

74 professional computer-work tasks across 7 domainsTask completion rateFrontier autonomous knowledge work

The current Terminal-Bench release keeps the 66 professional computer-work tasks of 4.0 and regrades every saved trial with verifiers hardened against reward hacking and overly strict checks.

66 professional computer-work tasksTask completion rate across 5 trials per taskFrontier autonomous knowledge work

A benchmark of AI agents completing expert-curated research workflows across the life, physical, Earth, mathematical, and engineering sciences.

70 expert-curated scientific research workflowsResolution rate across 3 trials per taskFrontier scientific research workflows

A multi-turn benchmark of emotional and social intelligence using synthetic personas and pairwise LLM judging.

Up to 120 multi-turn persona scenarios per modelPairwise EloEmotional and social intelligence

A Markov Studios computer-use benchmark that asks agents to produce 2D drawings and 3D models in AutoCAD.

21 2D drawing tasks and 29 3D modeling tasksTask completion rate at a 75-point rubric thresholdProfessional CAD computer use

A display-only Elo rating from blinded comparisons of multi-file web applications built by coding agents.

Multi-file web application developmentElo from blinded human preferencesAgentic frontend development

An independently evaluated professional-work benchmark reported as Elo.

Professional knowledge-work tasksEloProfessional work

The strict pass@1 score on Artificial Analysis's AutomationBench-AA, where a business-workflow task counts only when it is fully completed.

End-to-end business-workflow automation tasksStrict pass@1Agentic automation

An independently evaluated automation benchmark from Artificial Analysis.

Business-process automation tasksTask success rateAgentic automation

An independently evaluated enterprise-operations benchmark from Artificial Analysis.

Enterprise operations workflowsTask success rateEnterprise agent operations

Legal-agent task success requiring every rubric criterion to pass and no material hallucination, averaged across three judges.

120 private legal tasks across 24 practice areasHallucination-Gated All-Pass RateProfessional legal work

An independently evaluated IT-operations benchmark from Artificial Analysis.

IT incident-response tasksTask success rateEnterprise IT operations

An independently evaluated Tau3 banking benchmark from Artificial Analysis.

Banking tool-use workflowsTask success rateAgentic banking workflows

A benchmark for agentic software engineering tasks executed in real terminal environments. Models must inspect files, run commands, edit code, and recover from errors over multi-step workflows.

Terminal-based software tasksInteractive CLI agent evaluationProfessional software engineering

Terminal-Bench 2.1 results, mostly provider-reported, stored separately from the Terminal-Bench 2.0 lane.

Terminal-based software-agent tasksInteractive task success rateProfessional software engineering

BrowseComp tests whether a browsing agent can find difficult, fact-seeking answers across the web. Its 1,266 problems use short answers that are easier to grade than an open-ended research report.

Research questions requiring browsingWeb search and evidence synthesisHard web research

Tool-augmented Humanity's Last Exam scores reported in DeepSeek-V4 thinking-mode evaluations.

Expert questions with tool usePass@1Frontier tool-augmented reasoning

An agentic real-world work-task evaluation reported as an Elo score in DeepSeek-V4 thinking-mode evaluations.

Agentic real-world work tasksEloProfessional agentic workflows

An Artificial Analysis normalized score for economically valuable tasks.

Economically valuable tasksNormalized scoreProfessional agentic workflows

A display-only Artificial Analysis agentic index.

Cross-benchmark agentic indexAggregated model scoreDisplay-only external reference

Artificial Analysis' document-output benchmark: professional tasks whose deliverable is a produced PDF, one of the ten components of its Intelligence Index v4.3.

Professional document-production tasksTask success rateProfessional knowledge work

Artificial Analysis' data-analysis benchmark, testing agents on spreadsheet and document work to answer the quantitative questions a business or data analyst faces day to day.

Spreadsheet and document analysis questionsTask success rateBusiness and data analysis

Artificial Analysis' implementation of the APEX-Agents benchmark for long-horizon professional-services agent tasks.

452 professional-services agent tasksPass@1Long-horizon workplace agent tasks

A game-environment benchmark that evaluates AI models in novel games covering strategic planning, resource management, spatial reasoning, cooperation, and theory of mind.

Novel game environmentsComposite game leaderboardAgentic coding and decision-making

OSWorld-Verified is the July 2025 repaired release of OSWorld's real-computer evaluation. It measures whether a model-agent system can finish desktop and web tasks from configured starting states, with success checked by execution-based evaluators.

369 real-world computer tasks (361 when eight Google Drive tasks are excluded)Execution-based interactive task successMulti-step desktop and cross-application workflows

OSWorld 2.0 evaluates long-horizon computer-use tasks. Published results use different task releases, metrics, and run settings, so scores appear by comparable protocol and unmatched rows stay unranked.

108 long-horizon computer-use workflowsInteractive computer-use evaluation; binary completion and partial-score metricsLong-horizon professional workflows

OSWorld 2.1 is the current release of OSWorld 2.0: the same 108 long-horizon computer-use workflows with the maintainers' later fixes to task files, assets, mocked websites, and evaluators. This page stores partial scores on the full task set.

108 long-horizon computer-use workflowsInteractive computer-use evaluation; partial scoreLong-horizon professional workflows

The official offline subset of OSWorld 2.1: 82 of the 108 long-horizon computer-use workflows, run with no internet access in the agent's virtual machine.

82 offline computer-use workflowsInteractive computer-use evaluation; partial scoreLong-horizon professional workflows

A cybersecurity task benchmark for evaluating defensive cyber workflows and vulnerability-oriented agent performance.

1,507 vulnerability analysis instancesVulnerability reproduction and PoC generationReal-world cybersecurity

A cybersecurity benchmark that measures whether an agent can turn raw threat-intelligence reports into working detection rules.

Threat-intelligence-to-detection-rule workflowsSuccess rateProfessional cyber threat detection

A cybersecurity benchmark of professional Capture the Flag tasks for measuring autonomous cyber agent capability and risk.

40 professional CTF tasksCybersecurity agent task completionProfessional cybersecurity

Artificial Analysis' composite index of cyber-defense capability: finding vulnerabilities in real codebases, reproducing and validating them, and patching them without breaking existing functionality. It averages CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA.

Three defensive cybersecurity evaluationsIndex score (0-100)Real-world vulnerability discovery and patching

Artificial Analysis' implementation of Collinear AI's CWE-bench, testing whether agents can patch real-world software vulnerabilities. One of the three components of the AA Cyber Index.

Real-world vulnerability patching tasksPass@1Automated vulnerability remediation

Artificial Analysis' implementation of Vercel's DeepsecBench, testing agents on finding security vulnerabilities in real web application code. One of the three components of the AA Cyber Index.

Vulnerability discovery in real codebasesPass@1Security code review

Artificial Analysis' implementation of Berkeley RDI's CyberGym-E2E: agents find a real memory-safety vulnerability in a C/C++ open-source project, write a proof-of-concept input that crashes it, and patch it without breaking the project's tests.

131 end-to-end vulnerability reproduction and patching tasksPass@1Real-world cybersecurity

Google DeepMind's benchmark of professional knowledge work where the key information sits in visual files such as architectural drawings, engineering schematics and medical scans. The agent has to deliver a finished report, spreadsheet, slide deck or marked-up drawing. Artificial Analysis runs it on a private 200-task set.

200 multimodal professional tasks across 10 industry verticalsClimb WSR (weighted rubric satisfaction rate, %)Expert multimodal knowledge work

OpenAI's benchmark of practical everyday tasks people want a personal assistant to handle, such as finding childcare, accessing government services, budgeting, shopping and caregiving.

169 everyday life tasks (caregiving, health, education, household administration, personal finance, shopping, government services)Average reward, unweighted across tasksAgentic personal-assistant work

A benchmark of real finance and accounting workflows built around spreadsheets: creating, editing and analysing workbooks the way an analyst or accountant would.

Spreadsheet-centric finance and accounting workflowsTask scoreEnterprise finance work

Lakera's open-source benchmark of how well the LLM inside an AI agent resists attacks such as system-prompt exfiltration, malicious code injection and tool-call manipulation. Higher means more attacks resisted.

Threat snapshots of attacks on agent backendsAttack resistance rateAgent security

Average refusal rate on the cyber-related prompts of JailbreakBench, StrongREJECT and AgentHarm, as measured by Mistral. Higher means the model refuses more malicious cyber requests.

Cyber-related harmful prompts from three safety benchmarksAverage refusal rateCyber misuse safety

A controlled benchmark for evaluating whether AI agents can extend vulnerability-triggering inputs into working exploits.

898 exploitation tasksWorking exploit generationAdvanced cybersecurity exploitation

An occupational agent benchmark for professional workflows that workers say they most want delegated to AI.

130 tasks across 35 occupationsAgentic workplace deliverablesProfessional multi-source workflows

A vision-language browsing benchmark for multimodal web research and tool-use workflows.

Multimodal browsing tasksVision-language web research evaluationMultimodal browser-agent

A computer-use benchmark for GUI task completion across the broader OSWorld task suite.

Computer-use tasksInteractive GUI evaluationBroad computer-use suite

A mobile GUI agent benchmark for completing Android app workflows and on-device tasks.

Android app workflowsInteractive mobile-agent evaluationComplex mobile task completion

A browser-agent benchmark for completing multi-step workflows on live websites.

Live website workflowsInteractive browser-agent evaluationMulti-step web navigation

A benchmark for tool-calling over Model Context Protocol integrations and external tools.

Tool-integrated agent tasksInteractive tool-calling evaluationAdvanced tool use

A Moonshot AI internal long-horizon agent benchmark for persistent professional coworking tasks.

17 professional scenarios, 610 evaluation pointsAverage pass rate across repeated OpenClaw runsLong-horizon agentic work

A human-verified edition of MCPMark for MCP tool use across Notion, GitHub, Filesystem, Postgres, and Playwright server environments.

MCP tool-use tasks across five server environmentsInteractive MCP task completionAdvanced tool use

A tool-use benchmark focused on selecting, sequencing, and completing tasks with external tools.

Multi-tool workflowsInteractive tool-calling evaluationAdvanced tool use

A verified tool-use benchmark variant for completing multi-step workflows with external tools.

Verified multi-tool workflowsInteractive tool-use scoreAdvanced tool use

An agent benchmark for completing automation workflows in reproducible task environments.

600 public automation tasksAgent task-completion scoreLong-horizon automation

An agent benchmark for completing business workflows across simulated applications and API endpoints.

600+ held-out workflows across 47 simulated applications and six business domainsTask pass rateLong-horizon automation

An agent benchmark reported in DeepSeek's V4 Flash 0731 launch comparison.

Agent tasksProvider-reported task scoreAdvanced agentic work

A professional-services agent benchmark covering long-horizon knowledge-work tasks.

Professional-services agent tasksAgent task-completion scoreLong-horizon professional work

A spreadsheet-focused benchmark for agentic analysis and editing workflows.

Spreadsheet analysis and editing tasksAgent task-completion scoreProfessional spreadsheet work

Moonshot AI's internal benchmark for presentation and deck-production workflows.

Internal presentation workflowsInternal evaluation scoreProfessional presentation creation

The Linux subset of Agents' Last Exam, an end-to-end agent task benchmark, as reported by Meta.

Linux agent tasks (subset size not published)Provider-reported task scoreAdvanced agentic work

The command-line subset of Agents' Last Exam v1.1: 86 of the release's 122 agent tasks.

86 CLI agent tasksProvider-reported task scoreAdvanced agentic work

Agentic investigation of biological datasets, reported as an overall score on the official v11 release.

Biological data investigationsOverall task scoreExpert scientific analysis

An investment-workflow benchmark covering complete tasks from screening to profitability analysis.

Investment research workflowsTask scoreProfessional finance

Legal-agent tasks from the public 1,251-task Harvey dataset, scored by the share of tasks that pass every rubric criterion.

1,251 public legal tasksAll-pass rateProfessional legal work

An agent benchmark run through its official OpenHands-based harness.

Agent tasks (v1 release)Task scoreAdvanced agentic work

Moonshot AI's internal benchmark for long-horizon management and decision-making in a simulated business environment.

Simulated business managementInternal evaluation scoreLong-horizon decision-making

Moonshot AI's internal benchmark for formula linkage, financial logic, and final workbook quality in real-world modeling tasks.

Spreadsheet financial modelsInternal evaluation scoreProfessional financial modeling

Moonshot AI's internal benchmark for auditable financial research under evidence and cutoff constraints.

Source-grounded research reportsInternal evaluation scoreProfessional financial research

Moonshot AI's internal arena for presentation generation on real-world queries, reported as Elo.

Real-world presentation requestsEloProfessional presentation creation

Moonshot AI's internal graphic-design benchmark using direct and reference-image few-shot prompting, reported as Elo.

Graphic design requestsEloProfessional graphic design

The hard split of Moonshot AI's internal factuality benchmark, run with tools and scored as the claim-level hallucination rate.

Hard internal work-task prompts with toolsClaim-level hallucination rateFactuality stress test

A Z.AI benchmark for OpenClaw-style agent workflows spanning information search, office work, data analysis, development and operations, automation, and security.

OpenClaw agent workflowsEnd-to-end agent benchmarkBroad productivity and operations workflows

This route is a sourced ledger for published τ²-bench results. Most current rows come from Artificial Analysis's telecom implementation, while named provider rows can use telecom, airline, retail, or aggregate setups.

Airline, retail, and telecom customer-service task setsPublished domain success or pass^k resultsDual-control customer-service workflows

An agentic browsing benchmark where models search the web, gather evidence, and answer list-style questions using browser tools.

Agentic browsing and list-answer questionsSearch / open / find browser-agent evaluationAgentic web research

τ²-bench Airline tests conversational agents on airline customer-service tasks governed by domain policy and database-changing tools.

Airline customer-service tasksDomain success under a published trial policyPolicy-constrained airline support workflows

An OpenClaw agent benchmark from Kilo that measures successful task completion across standardized real-world agent workflows.

23 OpenClaw agent tasksAverage success rate from official runsLong-horizon agent workflows

A holistic coding-agent benchmark that evaluates AI agents across issue resolution, frontend work, greenfield development, testing, and information gathering.

SWE-bench Verified, SWE-bench Multimodal, Commit0, SWT-bench Verified, and GAIAMacro-average across five coding-agent categoriesReal-world software engineering agent tasks

A Scale SWE-Atlas software-engineering agent benchmark focused on refactoring tasks.

SWE-Atlas refactoring tasksRefactoring score with confidence intervalsReal-world software-engineering agent tasks

Tests whether coding agents can find and fix multiple bugs across a repository without issue descriptions, while preserving existing behavior.

4,068 reference bugs across 100 repositories in 22 languagesMicro-average percentage of bugs resolved with a repository regression gateRepository-wide bug discovery and repair

Tests long-context agents as tool environments grow to a 256K-token environment description.

Agent tasks with controlled tool-environment growth

Tests whether coding agents can complete long-horizon, whole-repository stack migrations while preserving the original program's behavior.

20 whole-repository stack migrationsMigration audit, frozen behavioral checks, and agentic verification6- to 30-hour autonomous repository migrations

Tests whether coding agents can improve the training algorithm inside an existing AI research codebase, then survive a sealed training run and held-out evaluation.

10 AI training-algorithm design tasksFour-hour code rewrite followed by sealed training and held-out evaluationEnd-to-end AI research and algorithm design

A benchmark for open-ended LLM inference optimization by AI agents. Agents receive a base model, one H100, and a fixed time budget to build a valid OpenAI-compatible inference server that improves serving speed.

4 inference-serving optimization scenariosTwo-hour autonomous CLI agent runOpen-ended ML systems engineering

A ByteDance Seed benchmark of 134 real-world, day-scale tasks that measures how autonomous agents learn from environment feedback over 12+ hour interaction horizons, spanning scientific and ML, systems and software engineering, optimization, knowledge, formal, and game domains.

134 tasks (51 public) across 6 domainsLong-horizon interactive agent evaluationDay-scale expert tasks

A function-calling benchmark for tool selection, schema adherence, and argument correctness.

Function-calling tasksTool invocation and schema evaluationAdvanced tool use

A lightweight machine-learning competition benchmark that measures whether models can iteratively train, evaluate, and improve ML systems in low-resource settings.

Low-resource ML competitionsAutonomous iterative ML optimizationAgentic machine learning

An OpenClaw-derived agent benchmark covering practical work and life tasks such as office document delivery, research, planning, and code maintenance.

OpenClaw-style real-world tasksAgent workflow evaluationBroad real-world agentic execution

A transparent real-world autonomous-agent benchmark with 300 human-verified tasks, 2,159 rubric items, and Pass^3 scoring across general, multi-turn, and native multimodal agent tasks.

300 tasks, 2,159 rubricsEnd-to-end autonomous-agent evaluation with Pass^3 scoringReal-world general, multi-turn, and native multimodal agent execution

An end-to-end autonomous scientific research benchmark with 40 tasks across 10 scientific domains, where agents receive related literature and raw data, then attempt to rediscover the hidden target paper.

40 tasks across 10 scientific domainsEnd-to-end autonomous research evaluation with RADS scoringScientific research re-discovery

Qwen's internal OpenClaw-style benchmark for measuring broad real-world agent performance across practical productivity and research tasks.

Real-world agent workflowsEnd-to-end agent evaluationBroad real-world agentic execution

A Qwen benchmark for artifact and webpage generation quality reported as an Elo-style rating.

Web artifacts and interactive deliverablesElo-style artifact benchmarkArtifact generation

τ³-bench is the current evolution of Sierra's tool-agent-user framework, adding corrected task releases and newer knowledge and voice evaluation modes alongside airline, retail, and telecom.

Corrected customer-service tasks plus knowledge and voice evaluation modesPublished domain or average success resultsLong-horizon, multimodal, and knowledge-aware tool use

An interactive real-world agent benchmark grounded in practical consumer-service tasks such as delivery, in-store consumption, and online travel workflows.

Interactive consumer-service agent tasksEnd-to-end interactive agent evaluationLong-horizon real-world workflows

A long-horizon planning benchmark that tests whether agents can optimize under explicit time, budget, and feasibility constraints.

Travel planning and constrained shoppingLong-horizon planning benchmarkConstrained agent planning

A Model Context Protocol task benchmark used in Qwen's launch tables to measure practical execution over MCP-style tools and integrations.

MCP-integrated tool tasksInteractive tool-use evaluationAdvanced MCP workflows

A broad research-agent benchmark for open-ended information gathering, synthesis, and answer construction across wide search spaces.

Open-ended research tasksMulti-source research evaluationBroad research-agent workflows

Qwen's internal benchmark for long-horizon professional work across computer science, finance, law, medicine, and other productivity domains.

Long-horizon professional workflowsProvider-run agent scoreCross-domain professional work

A mobile-use agent benchmark for completing interactive tasks in smartphone environments.

Interactive mobile-device workflowsMobile agent task scoreLong-horizon mobile computer use

GAIA evaluates AI models on real-world tasks that are conceptually simple for humans but require multi-step reasoning, web browsing, tool use, and multimodal understanding for AI. Tasks span three difficulty levels and test practical assistant capabilities rather than academic knowledge.

466

Original TAU-bench tests an agent's tool use and policy compliance in simulated airline and retail conversations. Its pass^k metric asks whether the agent succeeds in every one of k repeated trials.

Airline and retail task sets in the archived 2024 releaseDomain-specific pass^1 through pass^4 task successPolicy-constrained, multi-turn customer service

WebArena tests whether a browser-agent system can complete 812 long-horizon tasks inside self-hosted replicas of functional websites. It checks the requested end state, so a result reflects the model, agent scaffold, browser interface, action budget, and evaluator together—not the base model alone.

812 long-horizon browser tasksEnd-state task successStateful multi-site browser work

WebArena-Verified is an audited release of the WebArena browser-agent benchmark. It rechecks task descriptions, reference answers, and evaluators, and replaces nondeterministic judging with deterministic checks where possible.

812 verified tasks; separate 258-task Hard subsetDeterministic end-state task successAudited stateful browser work

A benchmark that evaluates AI agents on multi-environment web challenges, testing navigation and task completion across diverse live web environments.

Web-agent tasksBrowser task completionOpen-web agent workflows

Vals AI benchmark for realistic financial analyst agent tasks across qualitative analysis, quantitative analysis, market work, comparables, precedents, earnings, disclosure, and modeling.

Financial analyst task categoriesMean score across repeated runsProfessional expert-task agent workflow

A quantitative-trading implementation benchmark that asks models to build backtesters under market-book liquidity and execution-delay constraints, then compares their outputs with a verifier.

3 quantitative-trading strategiesBacktester implementation scored by mean absolute errorMarket simulation and quantitative coding

A display-only provider-table GDPval rubric score for economically valuable work tasks.

Economically valuable work tasksRubric scoreProfessional agentic workflows

A display-only provider benchmark for finance-oriented tool-use and agent workflows.

Finance and banking tool-use tasksTask success rateProfessional finance-agent workflows

An index of agentic capability from Nous Research. It averages pass@1 scores on four suites, Hermes Bench, TerminalBench 4, TerminalBench Science and SkillsBench, with every run inside the Hermes Agent harness, and reports the mean cost per task beside the score.

Four suites: Hermes Bench (150 tasks), TerminalBench 4 without its 4 GPU tasks, TerminalBench Science and SkillsBenchMean pass@1 score across four suites, with mean cost per taskFrontier agentic

Tests agents on 150 tasks across 25 categories, including Hermes skills, research, diagrams and art, memory, tool use and safety. Each agent starts in a workspace with real files, some tasks add follow-up turns, and the grader checks the files and state left behind.

150 tasks across 25 categoriespass@1 scoreFrontier agentic

Tests whether a computer-use agent can learn a real accounts-payable job on the job, processing 100 vendor bills in a company ERP system with only the handbook, historical records, and mentor feedback a new hire would get.

100 vendor bills processed in sequence inside a simulated construction companyCumulative success rate over 100 billsLong-horizon computer use with offline and online continual learning

Tests whether a coding agent can explore a 256-room 3D puzzle world and collect its 100 hidden gems, using box-pushing puzzles, camera rotation, and long-horizon planning.

100 hidden gems across 256 rooms in one open worldGems collected out of 100, with rooms visited as a second scoreLong-horizon spatial planning, with single puzzles that can take over one hundred moves

A function-calling benchmark for tool selection, schema adherence, and argument correctness, covering single-turn, parallel, irrelevance and multi-turn subsets.

Function-calling tasksTool invocation and schema evaluationAdvanced tool use

Measures how reliably code-review agents identify useful issues in real pull requests while avoiding false positives.

219 pull requests across 187 repositories and 19 languagesGrounded precision, recall, and F1; augmented metrics

A Ramp Labs public leaderboard for long-horizon accounting agent work. BenchLM shows it as display-only reference data and excludes it from model rankings.

137 accounting tasks · 3 attempts per taskMean criterion-level score across three attemptsLong-horizon accounting agent evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

33 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

37 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

23 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

ToolComp-Chat: 485-prompt benchmark subset that needs models to chain a Python interpreter and web search calls to reach a final answer. BenchLM shows it as display-only reference data, excluded from rankings. Scale has not added a row to this board since March 2025.

ToolComp prompts, Chat subsetAccuracy (%)External agent and model evaluation

ToolComp-Enterprise: the subset of Scale’s 485-prompt tool-use benchmark that needs models to chain calls across 11 enterprise tools. BenchLM shows it as display-only reference data, excluded from rankings. Scale has not added a row to this board since March 2025.

ToolComp prompts, Enterprise subsetAccuracy (%)External agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

21 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A deep-research agent benchmark that pairs realistic, domain-diverse prompts with more than 2,500 expert-written rubrics scoring factual grounding, reasoning soundness and clarity of long-form, evidence-backed answers.

Deep-research prompts with 2,500+ expert-written rubricsRubric compliance (%)Open-ended deep research and long-form synthesis

The hardest task of FinSearchComp, an open financial search and reasoning benchmark for end-to-end agents: multi-step investigations over historical market and company data that mirror a financial analyst's research workflow.

Complex historical financial investigation questions (T3 subset of 635 total)Accuracy (%)Multi-step financial search and reasoning

Tests agents controlling a simulated robot arm and humanoid through motor commands, sensor readings, and camera images.

24 simulated robot-control tasksMean task score out of 100Real-time perception, planning, and motor control

SciUniverse Level 1 tests whether models carry out scientific work across chemistry, biology, and materials science. Its 92 tasks include physical experiments, simulated lab work, and measurement analysis, with Pass@1 averaged equally across 17 task families.

92 tasks in 17 task familiesScientific work through a facility harness; Pass@1Level 1 scientific tasks

A 29-model comparison of how well models use Parallel Search Fast and Extract to answer web research questions.

100 questions each from DeepSearchQA, HLE, and WISEREqual-weight Search Intelligence Score (0–100)Multi-step web research and specialist questions

Vals AI comparison of model routers with single models on long agentic work in finance, coding, law and tax, using the same agents, tasks and graders as the Vals Index.

About 1,250 agentic tasksAccuracy, cost per test and durationLong tool-using agent work in finance, coding, law and tax

Vals AI composite of agentic legal work: legal research, drafting and reviewing documents, and tax law, with practice areas weighted by their share of U.S. law-office revenue.

3 legal benchmarks across 8 practice areas plus taxRevenue-weighted accuracyAgentic legal research, document drafting and review, and tax research

Vals AI comparison of the same open-weight model served by different inference providers, against the model creator's own API, on agentic finance, coding and legal tasks.

165 agentic tasks from three Vals Index benchmarksAccuracy, cost per task and speed by model and providerLong tool-using agent work in finance, coding and law

Vals AI composite of agentic finance work: analyst research on company filings, financial modeling in Excel and corporate tax research, weighted by each sub-sector's share of U.S. GDP.

3 finance benchmarksGDP-weighted accuracy across three benchmarksAgentic analyst research, Excel financial modeling and corporate tax research

Vals AI composite of agentic science work: investigating biology datasets, recovering hidden mechanisms through experiments, and running research workflows in a terminal.

3 science benchmarksMean accuracy across three benchmarksAgentic biology, mechanism discovery and research workflows

Tests autonomous AI research on compression, language-model training, judge-harness engineering, and post-training under fixed compute and time budgets.

4 autonomous research tasksMean reference-anchored log score, expressed as a percentageAI research under fixed compute and time budgets

A Vals AI comparison of native provider search and Exa across finance-analysis and legal-research tasks.

Finance Agent Benchmark v2 and Legal Research Benchmark tasksAccuracy by model and search-tool combinationProfessional web research with controlled search-tool variants

A Vals AI agent benchmark that gives each system five days to build and run a space program in Kerbal Space Program.

30 progressively harder Kerbal Space Program missionsMission-ladder progress with partial creditLong-horizon autonomous computer use and planning

A Vals AI computer-use benchmark in which agents play six commercial video games with a keyboard and mouse.

Six commercial video-game control tasks, including held-out gamesOverall score with game-level task splitsKeyboard-and-mouse computer use in real-time games

An archived Vals benchmark covering retrieval, numerical reasoning, financial modeling, market analysis, earnings, trends, and adjustments.

11 financial analyst task viewsAccuracy with task-level breakdownsProfessional financial analysis

A contamination-resistant agent benchmark for reverse engineering real-world binaries.

Real-world binary reverse-engineering tasksFully solved rate and capability scoreAgentic reverse engineering

A Vals-hosted view of Terminal-Bench 1.0 with easy, medium, and hard task splits.

Terminal tasks split by easy, medium, and hard difficultyAccuracy scoreTerminal-agent execution

A contamination-controlled benchmark that asks agents to reverse engineer compiled binaries without source access and satisfy six task-specific objectives per challenge.

262 binary reverse-engineering instancesFully solved challenge rate (pass@1)Binary reverse engineering

A cybersecurity benchmark for evaluating LLM agents on full-control V8 exploit synthesis using 16 measured exploit capability flags.

V8 exploit synthesis runsCapability coverage percentage over 16 flagsBrowser exploitation and cybersecurity

Aikido Security's live benchmark of how well AI models rediscover real vulnerabilities. Each model runs three times on 32 recently disclosed CVEs in open-source repositories, and the score is the share it finds in at least one run (pass@3 recall).

32 recently disclosed CVEs in real open-source repositoriesThree runs per model in Aikido's code-analysis harness (30 turns, no internet)Rediscovering real vulnerabilities from source, including missing checks and long call chains

Letta's benchmark of how good the memory is that an AI model writes from past agent work. The model studies a corpus of earlier agent transcripts and writes memory files; the score is the share of 15 memory-dependent coding and knowledge-work tasks that a task agent then fully passes with that memory, averaged over three runs.

15 memory-dependent coding and knowledge-work tasks adapted from DeepSWE, SkillsBench, AutomationBench, APEX-Agents and EnterpriseOps-GymMemory written from a corpus of past agent transcripts in Letta Dream (6-hour limit, no internet), then used by a task agent in Letta Code; three runsLearning task conventions, user preferences and environment facts from past transcripts that the task prompt leaves out

Mercor's benchmark of AI agents on 26 long, open-ended engineering tasks in science, software, ML and engineering, built by domain experts. Each model makes 8 attempts per task, of 4 to 18 hours each, and the score is the mean task score on a scale from the starting point (0) to the best result the task allows (100).

26 expert-built engineering tasks in 4 domains (science, software, ML, engineering)8 attempts per task in the Harbor agent harness, 4 to 18 hours each, web search over an October 1, 2026 copy of the webLong-horizon engineering work: kernels, data systems, scientific forecasting, CAD and structural and aircraft design

Valen Labs' benchmark of AI agents doing machine learning work in computational biology: 25 tasks in antibody development, designed binders, protein engineering, structural biology and functional genomics. Each agent builds a model from experimental data in an offline sandbox, and its prediction script is scored on held-out measurements, from 0 (a simple baseline) to 100 (a perfect answer), over three runs per task.

25 machine learning tasks in computational biology across antibodies, designed binders, protein engineering, structural biology and functional genomicsOffline agent sandbox (4 to 32 CPUs, no GPU, about 6 hours); the submitted prediction script runs twice on held-out data; 3 runs per task, 75 per modelBuilding and validating models on messy biological data that generalize to held-out antigens, families, cell types and targets

Concept's benchmark of how well an AI agent understands the company it works in. Agents answer 205 questions drawn from one month of Concept's own Slack history: who wrote a message, who was tagged and who sent each reply. The score is overall accuracy.

205 questions from one month of Concept's Slack: 100 Authorship, 50 Mentions and 55 Reply matching panelsReplay of the Slack history in sandbox workspaces; agent products at default settings, API models in a shared Slack harnessKnowing who owns what, who works with whom and how each teammate thinks, from Slack alone

Scale AI's benchmark of whether AI agents stay safe while carrying out ordinary workplace requests. Agents handle 110 benign tasks in nine simulated workspaces, where the risk lies in how the work is done, such as leaking private details in a calendar invite. The score is safety pass^5: the share of tasks on which all five attempts were safe.

110 benign workplace requests across nine simulated workspaces: 70 communication tasks, 34 file operations and 6 workspace application changesAgent runs in simulated workspaces with MCP tools; five attempts per task, judged by GPT-5.6 Sol against each task's rubricRespecting boundaries the requester never stated, from workspace records and social norms

An enterprise-operations agent benchmark from Surge AI set in a simulated company (CoreCraft), where agents complete multi-step business tasks across internal tools and records.

Enterprise operations tasks in a simulated companyTask success (%)Multi-step enterprise tool use

Epoch AI's SpireClimb has models play Slay the Spire and Slay the Spire 2 at each game's maximum difficulty through a computer-use harness, as an end-to-end test of computer use, long-horizon reasoning and tactical play.

60 Slay the Spire 1 runs (Ascension 20, Heart kill) and 75 Slay the Spire 2 runs (Ascension 10) per modelWin rate, averaged across the two games, computer-use harnessMaximum game difficulty, long-horizon computer use

Epoch AI's EBR-bench tests whether models get better at an unfamiliar game with practice: they play the campaign card game Earthborne Rangers repeatedly, take notes, and are scored on later playthroughs.

Ten playthroughs of an Earthborne Rangers campaign segment per sample (eight learning, two scored)Long-horizon game play with notes across playthroughsLearning an unfamiliar game over multi-hour playthroughs

Epoch Automation Reports runs AI agents on open-ended tasks taken from Epoch AI's own work, such as graphic design, data insights and AI data center research, and has human graders score the output against Epoch's internal standards.

11 open-ended work tasks in five categories from Epoch AI's own workAgent harness at highest reasoning setting, human rubric gradingOpen-ended professional research and design work

A financial due-diligence agent benchmark from Thoughtful, scoring how well a model researches and analyzes a company the way an investment analyst would.

Financial due-diligence research tasksScore (%)Agentic financial analysis

The finance track of DAYJOB, a benchmark of end-to-end professional finance work that a model must complete as a full job task rather than a single question.

Professional finance job tasksTask success (%)Long-horizon finance work

OpenAI's benchmark of whether a model can build a working exploit for a known vulnerability and use it against a remote target, on real applications such as GitLab, PostgreSQL and Jenkins deployed in a controlled environment.

Known (N-day) vulnerabilities in deployed applications such as GitLab, PostgreSQL and JenkinsN-day target success rateRemote exploit development

OpenAI's offensive web penetration-testing evaluation: the model probes web applications without access to their source code and is scored on the share of planted vulnerabilities it finds.

Black-box web application penetration testsVulnerability recall (Web Blackbox score)Web vulnerability discovery

OpenAI's internal port of ExploitBench restricted to V8 and Chrome vulnerabilities disclosed in June to August 2026, to limit contamination from historical exploits.

Recently disclosed V8 and Chrome vulnerabilities (June–August 2026)Mean arbitrary-code-execution success rateBrowser exploitation

A hardened variant of ExploitGym that adds address-space randomization and stack canaries to userspace targets, enables the V8 sandbox and uses stricter kernel configurations where supported.

ExploitGym tasks with stronger exploit mitigationsIntended-exploit success rateExploitation against mitigations

An alignment test for cyber agents: hard ExploitGym tasks are paired with an easy-to-exploit auxiliary service (such as a package registry or object store) that would reveal the flag but is outside the authorized scope. Lower scores mean the model stays in scope more often.

Hard ExploitGym tasks with an out-of-scope honeypot serviceOut-of-scope exploitation rate (lower is better)Cyber-agent scope adherence

Moonshot AI's internal benchmark of expert-curated, source-code-based vulnerability discovery and exploitation tasks, most reproducing recently disclosed vulnerabilities with mitigations enabled.

Vulnerability exploitation tasksPass@3Offensive security research

Number of advanced cyber-range challenges solved in the joint NIST CAISI and UK AISI preliminary evaluation.

41 advanced cyber-range challengesChallenges solvedAdvanced cyber operations

Average step reached on a 32-step long-horizon cyber range.

32-step long-horizon cyber rangeAverage step reachedLong-horizon cyber operations

Share of runs that completed the 32-step cyber range within the 100-million-token limit.

10 long-horizon runsCompletion rateLong-horizon cyber operations

Cybersecurity benchmark for agentic vulnerability analysis and exploit-oriented security tasks.

Security engineering tasksSuccess rateAdvanced cybersecurity

Independent evaluation of AI agents against vulnerable real-world systems in dynamic environments.

197 dynamic cyber tasksTasks solvedEasy through elite cyber operations

Average success rate across realistic, long-horizon cybersecurity scenarios.

11 cyber scenariosAverage success rateLong-horizon cybersecurity

Number of CyScenarioBench scenarios completed successfully in at least one run.

11 cyber scenariosScenarios solved at least onceLong-horizon cybersecurity

Irregular's domain-level evaluation of network attack simulation capability.

Atomic cyber tasksDomain averageNetwork attack simulation

Irregular's domain-level evaluation of vulnerability research and exploitation capability.

Atomic cyber tasksDomain averageVulnerability research and exploitation

Irregular's domain-level evaluation of cybersecurity evasion capability.

Atomic cyber tasksDomain averageCybersecurity evasion

Share of SCONE smart-contract vulnerabilities exploited on the 12-task post-cutoff set.

12 post-cutoff smart-contract vulnerabilitiesBest@8 exploit success rateSmart-contract exploitation

Simulated value captured across the SCONE post-cutoff smart-contract set.

12 post-cutoff smart-contract vulnerabilitiesSimulated USD millions, Best@8Smart-contract exploitation

Share of patched Firefox 147 JavaScript-engine targets for which the model produced a working arbitrary-code-execution exploit.

250 Firefox 147 vulnerability trialsWorking arbitrary-code-execution rateBrowser exploit development

Share of evaluated OSS-Fuzz entry points where the model produced at least a crash.

Approximately 830 OSS-Fuzz entry pointsAny-crash rateVulnerability discovery

Share of evaluated OSS-Fuzz entry points where the model achieved a write primitive or stronger result.

Approximately 830 OSS-Fuzz entry pointsWrite-primitive-or-higher rateVulnerability exploitation

OpenAI's black-box, no-source variant of CVE-Bench v1 across 40 critical vulnerabilities.

40 critical CVEsPass@1 over three rolloutsBlack-box vulnerability exploitation

A benchmark for agentic professional workflows with verifiable success criteria, reporting pass rates and partial scores for model plus agent-harness rows.

152 ALE-V1 professional workflow tasks across 13 top-level domainsPass rate, partial-credit score, cost, token, and duration metadataReal-world agentic workflows

An agentic coding benchmark where models use a TypeScript SDK to play a RuneScape-like environment and optimize skill-training performance.

16 RuneScape skill-training tasksAverage log XP-rate scoreAgentic gameplay automation

An independent agentic-intelligence evaluation from Toloka using private simulated workflows and a pass^5 metric.

Private simulated enterprise workflowspass^5 arena scoreAgentic workflow reliability

Can autonomous agents craft PoC inputs that trigger OSS-Fuzz vulnerabilities—and stop crashing after the fix?

OSS-Fuzz PoC and patch-verification tasksAccuracy scoreAutonomous cybersecurity exploit reproduction

Independent agent runs on biological data investigations, split by human-solvable and human-difficult tasks.

End-to-end bioinformatics investigations from raw dataAverage task accuracy across three runsAgentic biological analysis

Agent evaluation on research-grade US tax questions using the Tax Agent Bench harness.

193 private US tax test questions across six categoriesCitation-adjusted all-pass accuracy; Weighted Pass separateResearch-grade US tax questions

Tests an agent's ability to complete legal work using documents, spreadsheets, presentations, and file-system tools

Legal agent work across documents, spreadsheets, presentations, and filesMean task pass rate across two judges; criteria pass rate separateProfessional legal workflow automation

Evaluating agents on legal research tasks across diverse areas of US law

US-law legal research tasksAccuracy scoreProfessional legal research

SkillsBench, built by BenchFlow, measures how much agent Skills (instruction folders an agent can load) help on 87 tasks in eight domains.

Agent skill-importance tasksAccuracy scoreAgent skill evaluation

Which model can make the most money playing poker?

Poker-playing agent trialsAccuracy scoreStrategic game-agent decision making

Can AI help people navigate SNAP benefits?

SNAP public-benefits navigation tasksAccuracy scorePublic-benefits policy navigation

Can AI help someone navigate a civil legal problem without a lawyer?

252 simulated legal-aid conversations across 8 topicsShare of expert rubric criteria metFirst-contact civil legal help with local referrals

Can AI coach a household through a real financial problem?

256 simulated coaching conversations across 7 financial areasShare of practitioner checklist items metHousehold financial coaching with follow-up questions

Tests scientific discovery by asking agents to infer undisclosed rules across 70 interactive text-based games.

70 interactive games across seven difficulty tiersExpected composite score with confidence intervalUnknown-rule discovery and interactive reasoning

Tests long-horizon coherence by asking agents to operate a simulated vending-machine business for one year.

One year of simulated autonomous business operation per runArithmetic mean final bank balance in USDLong-horizon agent coherence and business operation

Can agents rediscover sealed mathematical mechanisms through bounded experiments?

Sealed mathematical mechanisms probed under an experiment budgetAccuracy scoreAgentic scientific discovery

Professional-services tasks across investment banking, management consulting, and corporate law. This table shows Mercor-run configurations for reference and is excluded from model rankings.

240 tasks in the version 1.1 revisionPass@1Source-specific evaluation

Month-end close tasks across ten simulated companies, worked through accounting software, spreadsheets and PDFs. This table shows Mercor-run configurations for reference and is excluded from model rankings.

100 held-out accounting tasksMean ScoreSource-specific evaluation

Finding difficult information through web research. This table shows Mercor-run configurations for reference and is excluded from model rankings.

100 Mercor tasks; 130 tasks in the separately reported original setPass@1Source-specific evaluation

Producing workplace deliverables from reference files. This table shows Mercor-run configurations for reference and is excluded from model rankings.

132 Mercor tasks; 213 tasks in the separately reported original setMean ScoreSource-specific evaluation

Completing tasks in command-line environments. This table shows Mercor-run configurations for reference and is excluded from model rankings.

99 Mercor tasks; 89 tasks in the separately reported original setPass@1Source-specific evaluation

Producing workplace deliverables from reference files. This table shows Mercor-run configurations for reference and is excluded from model rankings.

213 source-reported tasksMean ScoreSource-specific evaluation

Writing research reports from web evidence. This table shows Mercor-run configurations for reference and is excluded from model rankings.

130 source-reported tasksMean ScoreSource-specific evaluation

Producing legal work across practice areas and contract tasks. This table shows Mercor-run configurations for reference and is excluded from model rankings.

1,749 source-reported tasksPass@1Source-specific evaluation

Negotiating and revising contracts across multiple turns. This table shows Mercor-run configurations for reference and is excluded from model rankings.

140 source-reported tasksTurn-Weighted Mean ScoreSource-specific evaluation

Handling simulated banking-support conversations with tools. This table shows Mercor-run configurations for reference and is excluded from model rankings.

97 source-reported tasksPass@1Source-specific evaluation

Long, end-to-end engineering projects: an agent works through a full project folder, finds the governing codes and physical limits, and produces deliverables a licensed engineer would sign off on.

56 hidden test tasks (85 in all)Rubric and check score, 0 to 100Long-horizon professional engineering

Taking messy genomics datasets through quality control, modeling and diagnostics to a verifiable answer. This table shows Mercor-run configurations for reference and is excluded from model rankings.

10 public problemsMean ScoreSource-specific evaluation

Financial research with tools, revised questions and rubrics, a new judge jury, and stricter treatment of hedged answers.

449 private test questions across nine analyst categoriesDealbreaker-gated partial credit; all-pass reported separatelyProfessional expert-task agent workflow

Excel financial modeling from a template or from scratch, with updated grading in v1.1.

Excel-based financial modeling tasksEqually weighted mean of seven category scoresProfessional finance modeling

FACTS Search V2 measures factual answering with a shared Brave Search tool. Overall, public, private, hedging, search-count, attempted-accuracy, and F1 measurements remain distinct.

Search-assisted factual questions; public and held-out setsTool-assisted answering with an automated grader

Agents play customer service staff, genuine customers, and identity thieves in a hidden-information authentication simulator.

Multi-agent customer-authentication simulation; task version 21Source-published percentage scoreAdversarial multi-agent communication

DAYJOB is Surge AI's test of long professional assignments: 80 finance tasks and 50 healthcare tasks that an agent must finish to an expert checklist. This is the combined 130-task board on Kaggle.

130 long-horizon tasks: 80 finance and 50 healthcareAgent deliverables graded all-or-nothing against expert rubricsLong-horizon professional work

The healthcare track of Surge AI's DAYJOB: 50 long clinical, pharmacy, payer and quality-assurance assignments that an agent must complete to an expert checklist.

50 long-horizon healthcare tasks across five subfieldsAgent deliverables graded all-or-nothing against expert rubricsLong-horizon clinical and healthcare operations work

HeyGen's Code2Video tests whether a model can turn a detailed creative brief into a finished motion-graphics video by writing code.

Creative briefs rendered as motion-graphics videoRelative rating plus a raw score against human referencesCode-driven motion design

Quesma's BabaIsBench has LLM agents play the puzzle game Baba Is You through a text interface, where a level is won by pushing word tiles around to rewrite its rules. The score here is the first-attempt pass rate on the eight intro levels.

8 intro levels of Baba Is You, three attempts per level (Stage 0); 15 further levels run once (Stage 1)Agent plays through a text interface in the Harbor frameworkRule-rewriting puzzles that need planning over many moves

Quesma's BinaryAudit asks AI agents to find backdoors and timebombs planted in compiled, stripped binaries of real server software, using the reverse engineering tools Ghidra and radare2. The score here is the share of detection runs that found the planted code.

19 backdoor and timebomb detection tasks on stripped binaries, three runs each (57 runs); 6 tooling and 8 clean-binary tasks reported alongsideAgent with Ghidra and radare2 in a terminal, Harbor frameworkReverse engineering without source code or symbols

FrontierPhysics, from BenchFlow, gives an AI agent a research project written by a working physicist and checks whether it gets the physics right and writes it up well enough to publish. It has 56 tasks across nine fields of physics, and each one is run three times.

56 research projects from 42 physicists across nine fields, three trials each (168 rollouts per setup)Agentic research in a sandbox: literature review, simulation or measurement, structured results and a written paperOpen-ended research projects that took their authors about 270 days on average

Post-training four base language models across seven weighted benchmarks, with ten hours and one H100 per run.

Post-training Qwen3 1.7B and 4B, SmolLM3 3B, and Gemma3 4BWeighted aggregate across four base models and seven benchmarksFrontier agent evaluation

The first seven-benchmark PostTrainBench board, kept by the source as an archived version.

Post-training four base models on seven benchmarksWeighted aggregate across four base models and seven benchmarksAutonomous ML engineering

Program-reconstruction hidden-test pass rate after the first of five sequential long-context episodes.

166 golden program-reconstruction tasksHidden-test pass rate after episode 1Long-context clean-room software engineering

An independently evaluated LiveCodeBench result from Artificial Analysis.

Contamination-resistant coding tasksPass rateCompetitive programming

Terminal-Bench 2.1 results, mostly provider-reported, stored separately from the Terminal-Bench 2.0 lane.

Terminal-based software-agent tasksInteractive task success rateProfessional software engineering

HumanEval tests whether generated Python functions pass unit tests on 164 handwritten problems. These scores describe function-level correctness; they do not measure repository work or a coding agent.

164 problemsPython function generationIntroductory to intermediate programming

A code-generation benchmark reported in DeepSeek-V4 base-model evaluations.

Code generation tasksPass@1Software engineering

Competitive-programming rating reported for DeepSeek-V4 thinking-mode evaluations.

Competitive programming contestsRatingElite competitive programming

A benchmark for agentic software engineering tasks executed in real terminal environments. DeepSeek reports it in the agentic section, while BenchLM also mirrors it in coding for models that publish it as a developer-task signal.

Terminal-based software tasksInteractive CLI agent evaluationProfessional software engineering

SWE-bench Verified tests patches for 500 human-filtered GitHub issues. A score measures how many tasks an evaluated model-and-agent setup resolved; the model name alone does not identify that setup.

500 verified issuesCode patch generationProfessional software engineering

A continuously updated software engineering benchmark by Nebius using fresh GitHub issues to avoid contamination. Models are evaluated 5 times per problem under a fixed ReAct scaffolding; the Resolved Rate (best pass@1) is reported.

Fresh GitHub issues (rolling window)Code patch generationProfessional software engineering

A continuously updated coding benchmark built from newly collected LeetCode, AtCoder, and Codeforces problems. Fresh problem windows reduce one contamination path, but results still need a release and setup check.

Continuously updated contest problemsCompetitive-programming evaluationCompetitive programming level

LiveCodeBench v6 is a named release slice used in provider comparison tables. Keeping it separate prevents v6 results from being mixed into older or rolling LiveCodeBench windows.

Fresh programming problemsProvider-published v6 competitive programming resultsCompetitive programming level

LiveCodeBench v5 is a named release and date-window slice. BenchLM keeps explicitly labeled v5 rows outside the rolling weighted lane.

July 2024 to May 2025 release windowProvider-published v5 competitive programming resultCompetitive programming level

This lane contains DeepSeek's LiveCodeBench Pass@1-COT results. The explicit metric and prompting label keeps them separate from generic and version-specific LiveCodeBench rows.

DeepSeek-V4 report evaluation windowPass@1-COT competitive programming resultsCompetitive programming level

A harder competitive-programming benchmark family built from Codeforces, ICPC, and IOI problems, with quarter-specific public leaderboards and difficulty-aware reporting.

Quarter-specific contest programming setsCompetitive programmingHigh-end contest programming

A repository-level Lean 4 proof engineering benchmark that measures whether a model can complete formal proofs and correctly define new mathematical concepts inside realistic FLT project pull requests.

FLT project pull requestsLean 4 repository task completionFormal verification / proof engineering

A long-horizon repository benchmark built to test realistic software engineering work. Its scores need a task-quality and setup check before they support a coding-agent decision.

1,865 repository problemsRepository task completionLong-horizon professional engineering

A Snorkel AI benchmark of senior-level software engineering tasks emphasizing under-specified feature work, bug/performance investigation, and taste-based correctness.

Senior-level repository tasksAgentic software-engineering evaluationProfessional senior engineering

An open coding benchmark of 23 hard behavioural-reconstruction tasks, run at every reasoning-effort level and scored on correctness and judged code quality.

23 behavioural-reconstruction coding tasksCombined score (correctness and judged code quality), best effort per modelProfessional multi-file software engineering

An open software-engineering benchmark built from real merged post-cutoff pull requests across Python, Rust, TypeScript, JavaScript, and Go repositories. Replaced by VulcanBench Frontier v4 in September 2026.

23 post-cutoff repository tasks in the v3 reportPass@1 with low, medium, and high effortProfessional multi-file software engineering

An app-level coding benchmark that asks DevEco Code configurations to build, extend, fix and migrate buildable OpenHarmony ArkTS applications.

App-development, bug-fix and migration tasks (count not published)Task completion through DevEco CodeEnd-to-end OpenHarmony application development and migration

An app-level coding benchmark that asks DevEco Code configurations to implement observable behavior in buildable OpenHarmony ArkTS applications. Replaced by OpenHarmony Bench v1.2 in November 2026.

153 app-development and bug-fix tasksTask completion through DevEco CodeEnd-to-end OpenHarmony application development

A post-cutoff software-engineering benchmark with hidden functional tests and regression guards, reported for vendor coding-agent harnesses.

38 validated post-cutoff repository tasksPass@1 with vendor coding-agent harnessesMid-band frontier software engineering

Cognition's 100-task software-engineering benchmark for whether coding agents produce mergeable, production-quality pull requests, scored for correctness, tests, scope, style, and maintainability through maintainer-authored rubrics.

100 private Main tasks (150 in Extended)Repository task completion with maintainer rubricsFrontier coding-agent quality

Cognition's 150-task Extended subset of the FrontierCode 1.1 software-engineering benchmark.

150 private software-engineering tasksRepository task completion with maintainer rubricsFrontier coding-agent quality

Artificial Analysis's own run of Cognition's FrontierCode 2.0 coding-agent benchmark, one of the nine evaluations in its Intelligence Index v5, judged on whether each pull request is mergeable.

132 private software-engineering tasksWeighted rubric score with blocking criteriaFrontier coding-agent quality

Cognition's second-generation coding-agent benchmark, scoring whether agents produce mergeable pull requests on larger, less fully specified tasks, with added emphasis on thoroughness such as fixing every instance of a bug.

132 private software-engineering tasksRepository task completion with maintainer rubricsFrontier coding-agent quality

IBM Research's benchmark for coding agents that migrate complete enterprise Java applications between Jakarta EE, Quarkus and Spring while keeping them working.

204 framework migration tasks across 34 enterprise Java applicationsPercent of smoke tests passed after migration (Kaggle)Whole-application framework migration

An 80-task software-engineering benchmark across eight repositories that tests whether autonomous IDE agents can explore, edit, run, and verify code changes end to end.

80 tasks across 8 repositoriesAutonomous IDE-agent task completion (pass@1)End-to-end software engineering

A six-task full-stack web-app benchmark that measures how much required functionality an AI builder or coding assistant delivers from one prompt without human code edits.

6 full-stack app-building tasksBest-of-three one-shot feature completionProduction-style full-stack application generation

A multilingual software-engineering benchmark for real-world code issue resolution across multiple programming languages.

Multilingual software-engineering tasksRepository task completionProfessional software engineering

A multimodal variant of SWE-bench that adds visual context such as screenshots and design mockups to software engineering issue descriptions.

Multimodal software engineering tasksCode patch generation with visual contextFrontier multimodal coding

Cursor's first-party benchmark for ambiguous, multi-file coding-agent tasks drawn from real Cursor sessions, currently on the 4.0 task set.

Long-horizon multi-file agentic coding tasksCursor agent-loop evaluationProfessional agentic software engineering

A multi-language software-engineering benchmark that measures repository-level bug fixing and implementation across more than one programming ecosystem.

Multi-language repo tasksRepository task completionProfessional software engineering

A repo-level code generation and full-project delivery benchmark spanning web, mobile, and simulation-style implementation tasks.

Full project delivery tasksRepository-level implementation benchmarkEnd-to-end software delivery

Vals.ai benchmark for evaluating whether models can build complete web applications from natural language specifications in a production-like development environment.

End-to-end web application buildsFull-stack app implementation benchmarkEnd-to-end software delivery

A cleanroom software-engineering benchmark where agents receive only a compiled executable and documentation, then must architect and implement a complete codebase that reproduces the original program's behavior.

200 program reconstruction tasksCleanroom executable reimplementationFull-repository software architecture

An ultra-long-horizon software-engineering benchmark with open-ended implementation, performance, and research tasks designed to challenge frontier coding agents.

17 ultra-long-horizon engineering and research tasksMean@5, best@5, average rank, and dominanceUltra-long-horizon frontier software engineering

A 34-task expansion of FrontierSWE for ultra-long-horizon engineering and research work that remains far from saturation.

34 ultra-long-horizon engineering and research tasksFive-trial mean task score (Mean@5), 0-100Ultra-long-horizon frontier software engineering

A code-comprehension benchmark covering production repositories across Go, Python, C, and TypeScript.

124 codebase questions across 11 repositoriesMean pass@1Production codebase comprehension

A blind-graded coding-agent benchmark with 105 planted bugs across two production TypeScript repositories.

105 planted bugs across two production TypeScript repositoriesStrict planted bugs fixedBlind production-repository bug finding and repair

A Moonshot AI internal coding-agent benchmark for realistic software-engineering tasks across mainstream programming languages and production technology stacks.

Realistic coding-agent tasksCoding-agent pass rateProduction software engineering

Moonshot AI's internal long-horizon coding-agent benchmark, the successor to Kimi Code Bench 2.0.

Internal long-horizon coding-agent tasksCoding-agent pass rateLong-horizon software engineering

An AI agent gets ten hours and one H100 GPU to post-train a small base model; the score is how well the trained models do on six benchmarks.

Post-training four base models on six benchmarksWeighted aggregate across four base models and six benchmarksAutonomous ML engineering

ProgramBench cleanroom reimplementation tasks scored as pass/fail: a task counts only when its ProgramBench score is at least 95%.

Cleanroom executable reimplementationShare of tasks at 95% or higherWhole-program reconstruction

A 30-task subset of MLS-Bench that evaluates whether AI systems can invent generalizable and scalable machine-learning methods.

30 machine-learning research tasksAgentic ML task evaluationML research and systems engineering

A research-reproduction benchmark that asks agents to recreate the contributions of AI papers from the paper alone.

AI research-paper reproductionLong-horizon agent evaluationFrontier autonomous research and engineering

Qwen's internal benchmark for building and rendering bilingual React projects across seven categories.

Bilingual React project constructionBradley-Terry/Elo ratingProduction frontend development

A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.

Natural language to repository tasksRepository understanding benchmarkSystem-level software comprehension

DeepSeek's internal full-stack coding-agent benchmark.

Internal full-stack coding-agent tasksProvider-reported scoreFull-stack software engineering

DeepSeek's internal hard coding-agent benchmark.

Internal hard coding-agent tasksProvider-reported scoreAdvanced coding-agent challenges

An open benchmark for AI coding agents on real-world React Native implementation tasks, emphasizing working app behavior, recommended architecture choices, and strict constraint adherence.

React Native app implementation tasksFramework-specific app development evaluationProduction mobile app engineering

A coding-agent benchmark for realistic React work, with rubrics that check production concerns such as performance, accessibility, correctness, and code quality.

51 production React tasksPass@1 weighted rubric scoreProduction frontend engineering

An agentic GPU-kernel benchmark that measures how much of the hardware roofline a model's correct, audit-clean kernels reach on six demanding CUDA and Triton problems.

6 GPU-kernel optimization problemsMean peak fraction of hardware roofline over valid cellsAgentic GPU systems engineering

A Vercel benchmark for AI coding agents on Next.js code generation and migration tasks, reporting pass@4, average execution time, and results with bundled documentation through AGENTS.md.

31 current Next.js code generation and migration tasksSuccess rate (pass@4), with and without bundled documentationFramework-specific web application engineering

A text-to-SQL benchmark over realistic warehouse-scale schemas, reported by Interfaze for model comparison.

Text-to-SQL queriesExecution accuracyEnterprise text-to-SQL

SciCode evaluates language models on generating code for realistic scientific research problems across 16 subfields of physics, math, chemistry, biology, and material science. Problems decompose into 338 subproblems requiring domain knowledge recall, scientific reasoning, and precise code synthesis. Based on real scripts from published research.

80

A display-only Artificial Analysis coding index.

Cross-benchmark coding indexAggregated model scoreDisplay-only external reference

A display-only Artificial Analysis leaderboard for coding-agent systems, combining agent harnesses, host models, and execution settings across software-engineering benchmarks.

Composite over DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnAAverage pass@1 indexReal-world coding-agent workflows

An Artificial Analysis SciCode score.

Scientific coding subproblemsTask success rateScientific programming

A display-only Artificial Analysis coding metric for agentic coding and terminal use on a harder Terminal-Bench slice.

Agentic coding and terminal tasksTask success rateProfessional software engineering

A display-only MiniMax provider benchmark for end-to-end coding-agent and product-building tasks.

End-to-end coding-agent tasksTask success rateFrontier coding-agent workflows

A display-only provider benchmark for generating or manipulating SVG outputs from natural-language requirements.

SVG generation and editing tasksTask success rateVisual coding and structured graphics generation

A display-only benchmark for difficult GPU kernel implementation and optimization tasks.

Hard GPU kernel coding tasksTask success rateSpecialized systems programming

Evaluates coding agents on 333 multimodal game-development tasks in Godot, spanning 2D graphics, 3D graphics, user interfaces, and gameplay logic.

333 tasks from 88 tutorialsPass@1 on the full task set with 95% confidence intervalsMultimodal game development in Godot 4.4.1

A systems and software-engineering benchmark from ByteDance Seed that evaluates agents on long-horizon edge tasks using time-budgeted learning curves rather than a single static pass rate.

Systems and software-engineering tasksTime-budgeted agent learning curvesLong-horizon engineering

Vals AI’s independent implementation of the LiveCodeBench code-generation benchmark, run under one fixed harness across the models it tracks.

Competitive programming problems (easy, medium, hard)Pass@1 accuracyFrontier coding

Vals AI’s independent run of the public SWE-bench issue set, reported by human time-to-fix bucket. Vals removed SWE-bench Verified from its index as saturated on 2026-05-04; this board is the standalone SWE-bench run.

Real repository issues by human time bucketResolved rateFrontier coding agents

An electrical-engineering agent benchmark maintained by atopile.

Electrical-engineering design tasksAgent task scoreProfessional electrical engineering

Hugging Face's open benchmark for generating valid and geometrically correct 3D mechanical parts.

49 CAD generation samplesValidity-gated CAD scoreProfessional mechanical CAD

A long-horizon software engineering benchmark from Datacurve for measuring frontier coding agents on original tasks drawn from active open-source repositories.

113 software engineering tasks across 91 repositories and 5 languagesPass@1 with confidence interval, cost, time, and token metadataLong-horizon software engineering

Scale’s SEAL coding board: expert reviewers compared model answers to 1,000 prompts across programming languages, debugging, optimization, and documentation tasks. BenchLM shows it as display-only reference data, excluded from rankings. Scale marked this board deprecated as of March 2025 and no longer adds models.

1,000 coding promptsElo-style ratingExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

28 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

28 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

28 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

28 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

18 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

18 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

24 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

17 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

24 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation

A 1,000-query web development benchmark that judges generated front-end apps by deploying them and letting an evaluator agent use them, scoring functionality and aesthetics without reference implementations or test suites.

1,000 web development queries (React setting)Benchmark score (0-100)Interactive web apps and static pages across three tiers

The 78 text-brief tasks of BlenderWorld, a benchmark where agents work inside Blender to turn real-world 3D creation requests into editable Blender projects and requested exports.

78 text-brief 3D creation tasks in BlenderTask score (0-100), half rubric and half visual qualityLong-horizon 3D modeling and scene building

A benchmark for vision-language model agents that write procedural code for 3D modeling software, turning a text description into an engine-ready 3D object. This page tracks the text-prompt track.

Text prompts to procedural 3D modeling codeBenchmark score (0-100)Code-level geometric reasoning and 3D software APIs

JavaScript issue resolution where bug reports depend on screenshots, evaluated by Vals AI with no agent internet access.

480 JavaScript issues with visual bug reportsResolved issue percentageVisual software issue resolution

BridgeBench publishes comparative model ratings across nine software-building axes, including reasoning, front end, back end, security, speed, and cost. Its scoring methodology is proprietary.

No public task countPublisher comparative ratingSoftware-building assessments

A long-horizon software engineering benchmark from Abundant AI with multi-hour tasks spanning library reproductions, full-stack product clones, and ML engineering.

20 multi-hour software engineering tasksTask resolution and trajectory reviewUltra-long-horizon software engineering

An agentic GPU kernel-optimization benchmark of 69 hard tasks, adapted by Thinking Machines Lab for consistent cross-model evaluation. Each model gets four independent attempts with a 10-hour budget, and the score is the area under the mean tasks-solved curve, out of a maximum of 690 task-hours.

69 hard GPU kernel-optimization tasksArea under the mean tasks-solved curve over 10 hours (task-hours, max 690)Long-horizon GPU kernel optimization

unLAP's MaxxBench takes 89 DeepSWE software engineering tasks and changes the instructions or starting code while keeping the core problem, to test whether coding agents adapt to new requirements or fall back on a memorized solution.

89 mutated DeepSWE feature-implementation tasks, two attempts per modelPass rate over 178 agent attempts, mini-swe-agent harnessLong-horizon software engineering under changed requirements

MirrorCode, built by Epoch AI with METR, asks AI agents to reimplement entire command-line programs from scratch without their source code, matching the original's output exactly on end-to-end and held-out tests.

30 reimplementation tasks (15 programs in two languages), three attempts eachAgentic black-box reimplementation, exact-output end-to-end testsDays-long software engineering without source code

A vulnerability-repair benchmark that asks models to patch real security vulnerabilities in open-source code and checks the fix with tests.

Real-world vulnerability patching tasksPatch success (%)Security vulnerability repair

An agentic coding benchmark that asks models to build a Game Boy Advance emulator from scratch and grades emulator behavior against procedural, audio, and gameplay tests.

27 emulator test casesOverall emulator scoreLong-horizon systems programming

Håvard Tveit Ihle's WeirdML v3 gives coding agents 11 hand-made machine-learning and data-analysis tasks with unfamiliar data, unspecified goals or very limited feedback, and scores how well and how efficiently they improve as they spend tokens.

11 hand-made ML and data-analysis tasks (15 configurations)Agent harness runs up to 50M tokens, token-curve scoringOpen-ended ML engineering with unfamiliar data and little feedback

Based on the International Olympiad in Informatics

International Olympiad in Informatics-style programming tasksAccuracy scoreOlympiad programming

Can language models rebuild programs from scratch?

Program reconstruction tasksAccuracy scoreCleanroom software engineering

Can a model extend one working web application across a long sequence of dependent requests?

Long chains of dependent web-application change requestsAccuracy scoreLong-horizon software delivery

Software integration and production-failure diagnosis with the Terminus-2 harness. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Integration and observability engineering tasksPass@1Source-specific evaluation

Software integration and production-failure diagnosis with the Inspect harness. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Integration and observability engineering tasksPass@1Source-specific evaluation

Answering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings.

50 Mercor tasks; 124 tasks in the separately reported original setPass@1Source-specific evaluation

Editing repositories to resolve verified GitHub issues. This table shows Mercor-run configurations for reference and is excluded from model rankings.

50 Mercor tasks; 500 tasks in the separately reported original setPass@1Source-specific evaluation

Answering questions about software repositories with an agent. This table shows Mercor-run configurations for reference and is excluded from model rankings.

124 source-reported tasksPass@1Source-specific evaluation

Solving scientific programming subproblems with executable checks. This table shows Mercor-run configurations for reference and is excluded from model rankings.

65 source-reported tasksMean ScoreSource-specific evaluation

Hardware design, verification, and comprehension tasks. This table shows Mercor-run configurations for reference and is excluded from model rankings.

517 source-reported tasksPass@1Source-specific evaluation

Resolving software issues on the DeepSWE v1.1 task set. This table shows Mercor-run configurations for reference and is excluded from model rankings.

113 source-reported tasksPass@1Source-specific evaluation

Reconstructing a program from its executable behavior and documentation. This table shows Mercor-run configurations for reference and is excluded from model rankings.

100 source-reported tasksPass@1Source-specific evaluation

Coding agents recreate nine reference web apps from an empty directory. Plato checks visual and functional behavior through end-to-end workflows.

9 reference web appsPublished full score (%)Long-running web application recreation

Reimplementing working programs in another language using the Pi agent harness and revised hidden tests.

120 CLI migration tasks and 10 COBOL tasks75% CLI and 25% COBOL accuracyProduction code migration

Quesma's OTelBench asks coding agents to add OpenTelemetry tracing, metrics and logging to small existing services in 11 programming languages. The score is the share of runs where the service then emits correct telemetry.

23 OpenTelemetry instrumentation tasks in 11 languages, three attempts each (69 runs)Agent edits a service in a Linux terminal, Harbor frameworkDistributed tracing and context propagation across services

Quesma's CompileBench hands an agent the unmodified source of an open-source project such as curl, jq or GNU coreutils and asks for a working binary. Tasks include static and musl builds, cross-compiling to Windows and ARM64, and reviving code from 2003.

15 build tasks on cowsay, jq, GNU coreutils and curl, three attempts each (45 attempts)Agent in a Docker Linux terminal builds a project from source; the binary is checkedDependency resolution, legacy toolchains, static linking and cross-compilation

Anthropic's internal set of unpublished research problems written by professional scientists, each about a month of research work, mostly in theoretical physics; BenchLM stores the 64-problem subset.

64 research problemsMean score over up to nine attemptsResearch-level science

Synthetic long-context benchmark of retrieval, multi-hop tracing, aggregation and question answering, scored at a 128K-token context.

Synthetic long-context tasksOverall accuracyLong-context retrieval and tracing

Synthetic long-context benchmark of retrieval, multi-hop tracing, aggregation and question answering, scored at a 64K-token context.

Synthetic long-context tasksOverall accuracy at 64KLong-context retrieval and reasoning

Tests whether a chat assistant recalls and reasons over information spread across long, multi-session conversation histories.

Questions over multi-session chat historiesOverall accuracyRecall across conversations

Scores how far a model advances 120 open research tasks across biology, chemistry, physics, astronomy, Earth science and engineering on a 0 to 4 progress scale, crediting supported partial work and honest statements of what remains unresolved.

120 research tasks, 20 in each of six STEM domainsAudited progress level 0 to 4 per task, refined to a fractional score; complete-run mean across all tasksOpen research targets; no run has reached a full-target success

Breadth-first-search graph traversal on 200 problems with context lengths from 256K to 1M tokens.

Breadth-first-search graph traversal on 200 problems with context lengths from 256K to 1M tokens.F1Frontier agent evaluation

ARC Prize fluid-intelligence benchmark using novel visual grid transformations.

Semi-private ARC-AGI-1 evaluation setVerified accuracyAbstract visual reasoning

A dataset for evaluating language models on multistep soft reasoning tasks specified in natural language narratives. Tests the ability to perform complex, structured reasoning.

Multi-step reasoningNarrative-based reasoningComplex reasoning tasks

A suite of 23 challenging tasks from the BIG-Bench collaborative benchmark where prior language models failed to exceed average human performance, even with chain-of-thought prompting.

23 tasksMixed reasoning tasksAdvanced reasoning

A reading-comprehension benchmark requiring discrete reasoning over paragraphs, reported in DeepSeek-V4 base-model evaluations.

Paragraph reasoning questionsF1Reading and numerical reasoning

A commonsense natural-language inference benchmark reported in DeepSeek-V4 base-model evaluations.

Commonsense completion questionsExact matchCommonsense reasoning

A commonsense coreference benchmark reported in DeepSeek-V4 base-model evaluations.

Coreference resolution questionsExact matchCommonsense reasoning

A Chinese Winograd Schema Challenge benchmark reported in DeepSeek-V4 base-model evaluations.

Chinese coreference questionsExact matchChinese commonsense reasoning

A word-chain reasoning benchmark that tests planning, recall, constraint following, and vocabulary depth by asking models to extend non-repeating edit-distance-1 chains.

50 starting words × 3 trialsDifficulty-weighted word-chain reasoningOpen-ended lexical planning

Tests whether model judgments rank argumentative critiques in the same order as expert human ratings across philosophy, AI alignment, and other concept-heavy texts.

224 texts and 608 within-text critique pairsAverage pairwise-ranking loss against expert ratingsFuzzy, expert-rated argumentative reasoning

A multi-step verifiable reasoning benchmark that evaluates whether models can solve pencil puzzles with unique solutions.

300 evaluation puzzlesDirect and agentic puzzle solve rateMulti-step verifiable reasoning

A long-context benchmark that measures whether models can actually use extended context windows for reasoning and retrieval.

Long-context tasksExtended-context retrieval and reasoningHard long-context

A long-context benchmark for memory, retrieval, and multi-round coherence over large contexts.

Long-context retrievalMulti-round long-context evaluationHard long-context

MRCR v2 slice focused on long-context retrieval at 64K-128K lengths.

8-needle retrieval tasksLong-context retrievalLong-context reasoning

MRCR v2 slice focused on very long contexts at 128K-256K lengths.

8-needle retrieval tasksVery-long-context retrievalVery long-context reasoning

MRCR v2 slice focused on retrieval across 256K-512K-token contexts.

100 eight-needle retrieval examplesMean sequence-matcher ratioVery long-context retrieval

MRCR v2 slice focused on retrieval across 512K-1M-token contexts.

100 eight-needle retrieval examplesMean sequence-matcher ratioMillion-token retrieval

Long-context graph traversal benchmark using breadth-first search tasks.

Graph traversal tasksLong-context graph reasoningAlgorithmic long-context reasoning

Long-context benchmark for recovering parent relationships inside graph tasks.

Graph parent-retrieval tasksLong-context graph reasoningAlgorithmic long-context reasoning

A million-token MRCR long-context retrieval benchmark reported in DeepSeek-V4 model evaluations.

Million-token retrievalLong-context retrieval MMRMillion-token long context

A million-token CorpusQA long-context question-answering benchmark reported in DeepSeek-V4 model evaluations.

Million-token corpus question answeringLong-context QA accuracyMillion-token long context

ARC-AGI-2 tests whether a system can infer a transformation from example grids and apply it to a new grid. Compare accuracy with the evaluation split, attempt limit, and cost attached.

Visual pattern completion and abstract reasoningGrid transformation puzzles with novel rulesExpert-level — hardest public reasoning benchmark

ARC Prize verified interactive reasoning results using a provider-specific stateful adapter, kept separate from Standard harness results.

An interactive successor to ARC-AGI-2 that evaluates whether an AI agent can learn unfamiliar task mechanics through action and feedback.

Interactive game-like tasks with hidden rulesAgentic task completion under a capped evaluation budgetFrontier agentic reasoning

A multistage statistical-reasoning benchmark for genomics and biological-data analysis agents.

129 genomics statistical-analysis workflowsEval-level pass rate across dependent analysis decisionsLong-horizon scientific reasoning

OpenAI and Ligo's benchmark of how well AI agents interpret raw wet-lab evidence, such as gel and blot images, colony plates, microscopy, sequencing files and instrument exports, spot experimental errors and recognize when the data cannot support a conclusion.

100 routine wet-lab analysis tasks (88 multimodal) across five domainsMean reward (% of rubric credit), five attempts per taskRoutine wet-lab data interpretation

A long-context retrieval benchmark that measures whether a model can recover relevant information embedded deep inside very long contexts.

Long-context retrievalNeedle-in-a-haystack recallLong-context memory

The hardest subset of GPQA featuring the most challenging graduate-level science questions. Sometimes reported separately from the standard GPQA benchmark.

Expert-level science questionsMultiple choice questionsGraduate-level scientific reasoning

An Artificial Analysis long-context reasoning evaluation.

Long-context reasoning tasksAccuracyLong-context reasoning

An open benchmark from Wisedocs, run by Artificial Analysis, measuring how well models reason over long, fragmented medical records with multi-document synthesis.

Long, fragmented medical-record reasoningAccuracyLong-context medical reasoning

An Artificial Analysis metric for research-level physics reasoning.

Research-level physics questionsAccuracyResearch-level physics reasoning

A benchmark that tests whether AI models challenge nonsensical, ill-posed, or logically flawed prompts instead of confidently generating incorrect answers. Measures the critical ability to push back on bad input.

Nonsensical and flawed prompts across multiple domainsPrompt challenge and refusal evaluationRobustness and critical reasoning

An automated evaluation framework using 1,000+ real-world user tasks covering reasoning, planning, coding, and creative writing. Highly correlated with Chatbot Arena human preference rankings.

1,024 real-world tasksReal-world task evaluationDiverse real-world scenarios

A Scale Labs public leaderboard mirrored as display-only reference data. It does not affect BenchLM rankings.

5 published rowsPublished Scale leaderboard scoreExternal agent and model evaluation