AI Benchmarks Directory

Agentic data-analysis tasks scored against per-task rubrics at a 980K-token budget.

Agentic data research and analysis tasksNormalized rubric scoreProfessional data analysis

BrowseComp accuracy from ten collaborating Opus 5 agents on a pre-release model and unreleased effort configuration.

BrowseComp web-research tasks10-agent team accuracyLong-horizon web research

Average coverage of required claims in answers produced during real-world MCP tool-use workflows.

Production-like multi-server MCP workflowsMean claim coverageReal-world tool use

Strict task success requiring every expert-written legal-work rubric criterion to pass.

1,235 legal-agent tasksAll-criteria pass rateProfessional legal work

Mean fraction of expert-written rubric criteria passed across legal-agent tasks.

1,235 legal-agent tasksMean criterion-pass rateProfessional legal work

Harvey AI's strict held-out task success rate requiring every rubric criterion to pass.

Harvey-held-out legal-agent tasksAll-criteria pass rateProfessional legal work

Harvey AI's mean criterion-level score on its held-out legal-agent evaluation.

Harvey-held-out legal-agent tasksMean criterion-pass rateProfessional legal work

Fraction of Toolathlon Verified tasks solved in at least one of three trials.

108 verified real-world tool-use tasksPass@3Long-horizon application tool use

Fraction of Toolathlon Verified tasks solved in all three independent trials.

108 verified real-world tool-use tasksAll-three-trials pass rateLong-horizon application tool use

Average assistant turns per Toolathlon Verified trajectory.

108 verified real-world tool-use tasksAverage trajectory lengthLong-horizon application tool use

A continuously maintained benchmark for difficult computer work, including coding, deep learning, finance, engineering, math, and science tasks.

74 professional computer-work tasks across 7 domainsTask completion rateFrontier autonomous knowledge work

The current Terminal-Bench release measures difficult computer work after recalibrating task resources, fixing unstable tasks, and removing tasks that no longer separate frontier systems.

66 professional computer-work tasksTask completion rate across 5 trials per taskFrontier autonomous knowledge work

A benchmark of AI agents completing expert-curated research workflows across the life, physical, Earth, mathematical, and engineering sciences.

70 expert-curated scientific research workflowsResolution rate across 3 trials per taskFrontier scientific research workflows

A multi-turn benchmark of emotional and social intelligence using synthetic personas and pairwise LLM judging.

120 multi-turn persona scenariosPairwise EloEmotional and social intelligence

A Markov Studios computer-use benchmark that asks agents to produce 2D drawings and 3D models in AutoCAD.

21 2D drawing tasks and 29 3D modeling tasksTask completion rate at a 75-point rubric thresholdProfessional CAD computer use

A display-only Elo rating from blinded comparisons of multi-file web applications built by coding agents.

Multi-file web application developmentElo from blinded human preferencesAgentic frontend development

An independently evaluated professional-work benchmark reported as Elo.

Professional knowledge-work tasksEloProfessional work

An independently evaluated automation benchmark from Artificial Analysis.

Business-process automation tasksTask success rateAgentic automation

An independently evaluated enterprise-operations benchmark from Artificial Analysis.

Enterprise operations workflowsTask success rateEnterprise agent operations

An independently evaluated legal-agent benchmark from Artificial Analysis.

Legal agent tasksTask success rateProfessional legal work

An independently evaluated IT-operations benchmark from Artificial Analysis.

IT incident-response tasksTask success rateEnterprise IT operations

An independently evaluated Tau3 banking benchmark from Artificial Analysis.

Banking tool-use workflowsTask success rateAgentic banking workflows

A benchmark for agentic software engineering tasks executed in real terminal environments. Models must inspect files, run commands, edit code, and recover from errors over multi-step workflows.

Terminal-based software tasksInteractive CLI agent evaluationProfessional software engineering

A provider-run Terminal-Bench 2.1 result stored separately from the repository's Terminal-Bench 2.0 lane.

Terminal-based software-agent tasksInteractive task success rateProfessional software engineering

A benchmark for web-browsing agents that must search, inspect sources, gather evidence, and return the correct answer to research-oriented questions.

Research questions requiring browsingWeb search and evidence synthesisHard web research

Tool-augmented Humanity's Last Exam scores reported in DeepSeek-V4 thinking-mode evaluations.

Expert questions with tool usePass@1Frontier tool-augmented reasoning

An agentic real-world work-task evaluation reported as an Elo score in DeepSeek-V4 thinking-mode evaluations.

Agentic real-world work tasksEloProfessional agentic workflows

A display-only Artificial Analysis normalized score for economically valuable tasks.

Economically valuable tasksNormalized scoreProfessional agentic workflows

A display-only Artificial Analysis agentic index.

Cross-benchmark agentic indexAggregated model scoreDisplay-only external reference

Artificial Analysis' implementation of the APEX-Agents benchmark for long-horizon professional-services agent tasks.

452 professional-services agent tasksPass@1Long-horizon workplace agent tasks

A game-environment benchmark that evaluates AI models in novel games covering strategic planning, resource management, spatial reasoning, cooperation, and theory of mind.

Novel game environmentsComposite game leaderboardAgentic coding and decision-making

OSWorld-Verified is the July 2025 repaired release of OSWorld's real-computer evaluation. It measures whether a model-agent system can finish desktop and web tasks from configured starting states, with success checked by execution-based evaluators.

369 real-world computer tasks (361 when eight Google Drive tasks are excluded)Execution-based interactive task successMulti-step desktop and cross-application workflows

A long-horizon computer-use benchmark covering realistic workflows across everyday and professional desktop tasks.

108 long-horizon computer-use workflowsInteractive computer-use evaluationLong-horizon professional workflows

A cybersecurity task benchmark for evaluating defensive cyber workflows and vulnerability-oriented agent performance.

1,507 vulnerability analysis instancesVulnerability reproduction and PoC generationReal-world cybersecurity

An external benchmark for evaluating whether coding agents can produce correct patches for real-world software vulnerabilities.

Real-world vulnerability patching tasksPass@1Automated vulnerability remediation

A cybersecurity benchmark that measures whether an agent can turn raw threat-intelligence reports into working detection rules.

Threat-intelligence-to-detection-rule workflowsSuccess rateProfessional cyber threat detection

A cybersecurity benchmark of professional Capture the Flag tasks for measuring autonomous cyber agent capability and risk.

40 professional CTF tasksCybersecurity agent task completionProfessional cybersecurity

A controlled benchmark for evaluating whether AI agents can extend vulnerability-triggering inputs into working exploits.

898 exploitation tasksWorking exploit generationAdvanced cybersecurity exploitation

An occupational agent benchmark for professional workflows that workers say they most want delegated to AI.

130 tasks across 35 occupationsAgentic workplace deliverablesProfessional multi-source workflows

A vision-language browsing benchmark for multimodal web research and tool-use workflows.

Multimodal browsing tasksVision-language web research evaluationMultimodal browser-agent

A computer-use benchmark for GUI task completion across the broader OSWorld task suite.

Computer-use tasksInteractive GUI evaluationBroad computer-use suite

A mobile GUI agent benchmark for completing Android app workflows and on-device tasks.

Android app workflowsInteractive mobile-agent evaluationComplex mobile task completion

A browser-agent benchmark for completing multi-step workflows on live websites.

Live website workflowsInteractive browser-agent evaluationMulti-step web navigation

A benchmark for tool-calling over Model Context Protocol integrations and external tools.

Tool-integrated agent tasksInteractive tool-calling evaluationAdvanced tool use

A Moonshot AI internal long-horizon agent benchmark for persistent professional coworking tasks.

17 professional scenarios, 610 evaluation pointsAverage pass rate across repeated OpenClaw runsLong-horizon agentic work

A human-verified edition of MCPMark for MCP tool use across Notion, GitHub, Filesystem, Postgres, and Playwright server environments.

MCP tool-use tasks across five server environmentsInteractive MCP task completionAdvanced tool use

A tool-use benchmark focused on selecting, sequencing, and completing tasks with external tools.

Multi-tool workflowsInteractive tool-calling evaluationAdvanced tool use

A verified tool-use benchmark variant for completing multi-step workflows with external tools.

Verified multi-tool workflowsInteractive tool-use scoreAdvanced tool use

An agent benchmark for completing automation workflows in reproducible task environments.

600 public automation tasksAgent task-completion scoreLong-horizon automation

An agent benchmark reported in DeepSeek's V4 Flash 0731 launch comparison.

Agent tasksProvider-reported task scoreAdvanced agentic work

A professional-services agent benchmark covering long-horizon knowledge-work tasks.

Professional-services agent tasksAgent task-completion scoreLong-horizon professional work

A spreadsheet-focused benchmark for agentic analysis and editing workflows.

Spreadsheet analysis and editing tasksAgent task-completion scoreProfessional spreadsheet work

Moonshot AI's internal benchmark for presentation and deck-production workflows.

Internal presentation workflowsInternal evaluation scoreProfessional presentation creation

A Z.AI benchmark for OpenClaw-style agent workflows spanning information search, office work, data analysis, development and operations, automation, and security.

OpenClaw agent workflowsEnd-to-end agent benchmarkBroad productivity and operations workflows

This route is a sourced ledger for published τ²-bench results. Most current rows come from Artificial Analysis's telecom implementation, while named provider rows can use telecom, airline, retail, or aggregate setups.

Airline, retail, and telecom customer-service task setsPublished domain success or pass^k resultsDual-control customer-service workflows

An agentic browsing benchmark where models search the web, gather evidence, and answer list-style questions using browser tools.

Agentic browsing and list-answer questionsSearch / open / find browser-agent evaluationAgentic web research

τ²-bench Airline tests conversational agents on airline customer-service tasks governed by domain policy and database-changing tools.

Airline customer-service tasksDomain success under a published trial policyPolicy-constrained airline support workflows

An OpenClaw agent benchmark from Kilo that measures successful task completion across standardized real-world agent workflows.

23 OpenClaw agent tasksAverage success rate from official runsLong-horizon agent workflows

A holistic coding-agent benchmark that evaluates AI agents across issue resolution, frontend work, greenfield development, testing, and information gathering.

SWE-bench Verified, SWE-bench Multimodal, Commit0, SWT-bench Verified, and GAIAMacro-average across five coding-agent categoriesReal-world software engineering agent tasks

A Scale SWE-Atlas software-engineering agent benchmark focused on refactoring tasks.

SWE-Atlas refactoring tasksRefactoring score with confidence intervalsReal-world software-engineering agent tasks

Tests whether coding agents can complete long-horizon, whole-repository stack migrations while preserving the original program's behavior.

20 whole-repository stack migrationsMigration audit, frozen behavioral checks, and agentic verification6- to 30-hour autonomous repository migrations

Tests whether coding agents can improve the training algorithm inside an existing AI research codebase, then survive a sealed training run and held-out evaluation.

10 AI training-algorithm design tasksFour-hour code rewrite followed by sealed training and held-out evaluationEnd-to-end AI research and algorithm design

A benchmark for open-ended LLM inference optimization by AI agents. Agents receive a base model, one H100, and a fixed time budget to build a valid OpenAI-compatible inference server that improves serving speed.

4 inference-serving optimization scenariosTwo-hour autonomous CLI agent runOpen-ended ML systems engineering

A ByteDance Seed benchmark of 134 real-world, day-scale tasks that measures how autonomous agents learn from environment feedback over 12+ hour interaction horizons, spanning scientific and ML, systems and software engineering, optimization, knowledge, formal, and game domains.

134 tasks (51 public) across 6 domainsLong-horizon interactive agent evaluationDay-scale expert tasks

A function-calling benchmark for tool selection, schema adherence, and argument correctness.

Function-calling tasksTool invocation and schema evaluationAdvanced tool use

A lightweight machine-learning competition benchmark that measures whether models can iteratively train, evaluate, and improve ML systems in low-resource settings.

Low-resource ML competitionsAutonomous iterative ML optimizationAgentic machine learning

An OpenClaw-derived agent benchmark covering practical work and life tasks such as office document delivery, research, planning, and code maintenance.

OpenClaw-style real-world tasksAgent workflow evaluationBroad real-world agentic execution

A transparent real-world autonomous-agent benchmark with 300 human-verified tasks, 2,159 rubric items, and Pass^3 scoring across general, multi-turn, and native multimodal agent tasks.

300 tasks, 2,159 rubricsEnd-to-end autonomous-agent evaluation with Pass^3 scoringReal-world general, multi-turn, and native multimodal agent execution

An end-to-end autonomous scientific research benchmark with 40 tasks across 10 scientific domains, where agents receive related literature and raw data, then attempt to rediscover the hidden target paper.

40 tasks across 10 scientific domainsEnd-to-end autonomous research evaluation with RADS scoringScientific research re-discovery

Qwen's internal OpenClaw-style benchmark for measuring broad real-world agent performance across practical productivity and research tasks.

Real-world agent workflowsEnd-to-end agent evaluationBroad real-world agentic execution

A Qwen benchmark for artifact and webpage generation quality reported as an Elo-style rating.

Web artifacts and interactive deliverablesElo-style artifact benchmarkArtifact generation

τ³-bench is the current evolution of Sierra's tool-agent-user framework, adding corrected task releases and newer knowledge and voice evaluation modes alongside airline, retail, and telecom.

Corrected customer-service tasks plus knowledge and voice evaluation modesPublished domain or average success resultsLong-horizon, multimodal, and knowledge-aware tool use

An interactive real-world agent benchmark grounded in practical consumer-service tasks such as delivery, in-store consumption, and online travel workflows.

Interactive consumer-service agent tasksEnd-to-end interactive agent evaluationLong-horizon real-world workflows

A long-horizon planning benchmark that tests whether agents can optimize under explicit time, budget, and feasibility constraints.

Travel planning and constrained shoppingLong-horizon planning benchmarkConstrained agent planning

A Model Context Protocol task benchmark used in Qwen's launch tables to measure practical execution over MCP-style tools and integrations.

MCP-integrated tool tasksInteractive tool-use evaluationAdvanced MCP workflows

A broad research-agent benchmark for open-ended information gathering, synthesis, and answer construction across wide search spaces.

Open-ended research tasksMulti-source research evaluationBroad research-agent workflows

Qwen's internal benchmark for long-horizon professional work across computer science, finance, law, medicine, and other productivity domains.

Long-horizon professional workflowsProvider-run agent scoreCross-domain professional work

A mobile-use agent benchmark for completing interactive tasks in smartphone environments.

Interactive mobile-device workflowsMobile agent task scoreLong-horizon mobile computer use

GAIA evaluates AI models on real-world tasks that are conceptually simple for humans but require multi-step reasoning, web browsing, tool use, and multimodal understanding for AI. Tasks span three difficulty levels and test practical assistant capabilities rather than academic knowledge.

466

Original TAU-bench evaluates a model-driven agent in simulated airline and retail customer-service conversations with domain tools, database state, and policy constraints.

Airline and retail task sets in the archived 2024 releaseDomain-specific pass^1 through pass^4 task successPolicy-constrained, multi-turn customer service

WebArena tests whether a browser-agent system can complete 812 long-horizon tasks inside self-hosted replicas of functional websites. It checks the requested end state, so a result reflects the model, agent scaffold, browser interface, action budget, and evaluator together—not the base model alone.

812 long-horizon browser tasksEnd-state task successStateful multi-site browser work

WebArena-Verified is an audited release of the WebArena browser-agent benchmark. It rechecks task descriptions, reference answers, and evaluators, and replaces nondeterministic judging with deterministic checks where possible.

812 verified tasks; separate 258-task Hard subsetDeterministic end-state task successAudited stateful browser work

A benchmark that evaluates AI agents on multi-environment web challenges, testing navigation and task completion across diverse live web environments.

Web-agent tasksBrowser task completionOpen-web agent workflows

Vals AI benchmark for realistic financial analyst agent tasks across qualitative analysis, quantitative analysis, market work, comparables, precedents, earnings, disclosure, and modeling.

Financial analyst task categoriesMean score across repeated runsProfessional expert-task agent workflow

A quantitative-trading implementation benchmark that asks models to build backtesters under market-book liquidity and execution-delay constraints, then compares their outputs with a verifier.

3 quantitative-trading strategiesBacktester implementation scored by mean absolute errorMarket simulation and quantitative coding

A display-only provider-table GDPval rubric score for economically valuable work tasks.

Economically valuable work tasksRubric scoreProfessional agentic workflows

A display-only provider benchmark for finance-oriented tool-use and agent workflows.

Finance and banking tool-use tasksTask success rateProfessional finance-agent workflows

Program-reconstruction hidden-test pass rate after the first of five sequential long-context episodes.

166 golden program-reconstruction tasksHidden-test pass rate after episode 1Long-context clean-room software engineering

An independently evaluated LiveCodeBench result from Artificial Analysis.

Contamination-resistant coding tasksPass rateCompetitive programming

An independently evaluated Terminal-Bench v2.1 result from Artificial Analysis.

Terminal-based agent tasksTask success rateAgentic software engineering

A provider-run Terminal-Bench 2.1 result stored separately from the repository's Terminal-Bench 2.0 lane.

Terminal-based software-agent tasksInteractive task success rateProfessional software engineering

A set of 164 handwritten Python function-generation problems. HumanEval is useful as a historical floor check, but BenchLM's current exact-source table is too small to support a broad frontier-coding verdict.

164 problemsPython function generationIntroductory to intermediate programming

A code-generation benchmark reported in DeepSeek-V4 base-model evaluations.

Code generation tasksPass@1Software engineering

Competitive-programming rating reported for DeepSeek-V4 thinking-mode evaluations.

Competitive programming contestsRatingElite competitive programming

A benchmark for agentic software engineering tasks executed in real terminal environments. DeepSeek reports it in the agentic section, while BenchLM also mirrors it in coding for models that publish it as a developer-task signal.

Terminal-based software tasksInteractive CLI agent evaluationProfessional software engineering

A curated, human-verified subset of SWE-bench that tests models on resolving real GitHub issues from popular open-source Python repositories like Django, Flask, and scikit-learn.

500 verified issuesCode patch generationProfessional software engineering

A continuously updated software engineering benchmark by Nebius using fresh GitHub issues to avoid contamination. Models are evaluated 5 times per problem under a fixed ReAct scaffolding; the Resolved Rate (best pass@1) is reported.

Fresh GitHub issues (rolling window)Code patch generationProfessional software engineering

A continuously updated coding benchmark built from newly collected LeetCode, AtCoder, and Codeforces problems. Fresh problem windows reduce one contamination path, but results still need a release and setup check.

Continuously updated contest problemsCompetitive-programming evaluationCompetitive programming level

LiveCodeBench v6 is a named release slice used in provider comparison tables. Keeping it separate prevents v6 results from being mixed into older or rolling LiveCodeBench windows.

Fresh programming problemsProvider-published v6 competitive programming resultsCompetitive programming level

LiveCodeBench v5 is a named release and date-window slice. BenchLM keeps explicitly labeled v5 rows outside the rolling weighted lane.

July 2024 to May 2025 release windowProvider-published v5 competitive programming resultCompetitive programming level

This lane contains DeepSeek's LiveCodeBench Pass@1-COT results. The explicit metric and prompting label keeps them separate from generic and version-specific LiveCodeBench rows.

DeepSeek-V4 report evaluation windowPass@1-COT competitive programming resultsCompetitive programming level

A harder competitive-programming benchmark family built from Codeforces, ICPC, and IOI problems, with quarter-specific public leaderboards and difficulty-aware reporting.

Quarter-specific contest programming setsCompetitive programmingHigh-end contest programming

A repository-level Lean 4 proof engineering benchmark that measures whether a model can complete formal proofs and correctly define new mathematical concepts inside realistic FLT project pull requests.

FLT project pull requestsLean 4 repository task completionFormal verification / proof engineering

A long-horizon repository benchmark built to test realistic software engineering work. Its scores need a task-quality and setup check before they support a coding-agent decision.

1,865 repository problemsRepository task completionLong-horizon professional engineering

A Snorkel AI benchmark of senior-level software engineering tasks emphasizing under-specified feature work, bug/performance investigation, and taste-based correctness.

Senior-level repository tasksAgentic software-engineering evaluationProfessional senior engineering

An open software-engineering benchmark built from real merged post-cutoff pull requests across Python, Rust, TypeScript, JavaScript, and Go repositories.

23 post-cutoff repository tasks in the v3 reportPass@1 with low, medium, and high effortProfessional multi-file software engineering

An app-level coding benchmark that asks DevEco Code configurations to implement observable behavior in buildable OpenHarmony ArkTS applications.

153 app-development and bug-fix tasksTask completion through DevEco CodeEnd-to-end OpenHarmony application development

A post-cutoff software-engineering benchmark with hidden functional tests and regression guards, reported for vendor coding-agent harnesses.

38 validated post-cutoff repository tasksPass@1 with vendor coding-agent harnessesMid-band frontier software engineering

Cognition's 100-task software-engineering benchmark for whether coding agents produce mergeable, production-quality pull requests, scored for correctness, tests, scope, style, and maintainability through maintainer-authored rubrics.

100 private Main tasks (150 in Extended)Repository task completion with maintainer rubricsFrontier coding-agent quality

Cognition's 150-task Extended subset of the FrontierCode 1.1 software-engineering benchmark.

150 private software-engineering tasksRepository task completion with maintainer rubricsFrontier coding-agent quality

An 80-task software-engineering benchmark across eight repositories that tests whether autonomous IDE agents can explore, edit, run, and verify code changes end to end.

80 tasks across 8 repositoriesAutonomous IDE-agent task completion (pass@1)End-to-end software engineering

A six-task full-stack web-app benchmark that measures how much required functionality an AI builder or coding assistant delivers from one prompt without human code edits.

6 full-stack app-building tasksBest-of-three one-shot feature completionProduction-style full-stack application generation

A multilingual software-engineering benchmark for real-world code issue resolution across multiple programming languages.

Multilingual software-engineering tasksRepository task completionProfessional software engineering

A multimodal variant of SWE-bench that adds visual context such as screenshots and design mockups to software engineering issue descriptions.

Multimodal software engineering tasksCode patch generation with visual contextFrontier multimodal coding

Cursor's current first-party benchmark for ambiguous, multi-file coding-agent tasks from real Cursor sessions.

Harder long-horizon agentic coding tasksCursor agent-loop evaluationProfessional agentic software engineering

A multi-language software-engineering benchmark that measures repository-level bug fixing and implementation across more than one programming ecosystem.

Multi-language repo tasksRepository task completionProfessional software engineering

A repo-level code generation and full-project delivery benchmark spanning web, mobile, and simulation-style implementation tasks.

Full project delivery tasksRepository-level implementation benchmarkEnd-to-end software delivery

Vals.ai benchmark for evaluating whether models can build complete web applications from natural language specifications in a production-like development environment.

End-to-end web application buildsFull-stack app implementation benchmarkEnd-to-end software delivery

A cleanroom software-engineering benchmark where agents receive only a compiled executable and documentation, then must architect and implement a complete codebase that reproduces the original program's behavior.

200 program reconstruction tasksCleanroom executable reimplementationFull-repository software architecture

A software-engineering benchmark for post-training infrastructure and implementation tasks, evaluated through the official Harbor implementation.

Post-training software-engineering tasksHarbor agent evaluationFrontier software engineering

An ultra-long-horizon software-engineering benchmark with open-ended implementation, performance, and research tasks designed to challenge frontier coding agents.

17 ultra-long-horizon engineering and research tasksMean@5, best@5, average rank, and dominanceUltra-long-horizon frontier software engineering

A 34-task expansion of FrontierSWE for ultra-long-horizon engineering and research work that remains far from saturation.

34 ultra-long-horizon engineering and research tasksFive-trial mean task score (Mean@5), 0-100Ultra-long-horizon frontier software engineering

A code-comprehension benchmark covering production repositories across Go, Python, C, and TypeScript.

124 codebase questions across 11 repositoriesMean pass@1Production codebase comprehension

A blind-graded coding-agent benchmark with 105 planted bugs across two production TypeScript repositories.

105 planted bugs across two production TypeScript repositoriesStrict planted bugs fixedBlind production-repository bug finding and repair

A Moonshot AI internal coding-agent benchmark for realistic software-engineering tasks across mainstream programming languages and production technology stacks.

Realistic coding-agent tasksCoding-agent pass rateProduction software engineering

A 30-task subset of MLS-Bench that evaluates whether AI systems can invent generalizable and scalable machine-learning methods.

30 machine-learning research tasksAgentic ML task evaluationML research and systems engineering

A research-reproduction benchmark that asks agents to recreate the contributions of AI papers from the paper alone.

AI research-paper reproductionLong-horizon agent evaluationFrontier autonomous research and engineering

Qwen's internal benchmark for building and rendering bilingual React projects across seven categories.

Bilingual React project constructionBradley-Terry/Elo ratingProduction frontend development

A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.

Natural language to repository tasksRepository understanding benchmarkSystem-level software comprehension

DeepSeek's internal full-stack coding-agent benchmark.

Internal full-stack coding-agent tasksProvider-reported scoreFull-stack software engineering

DeepSeek's internal hard coding-agent benchmark.

Internal hard coding-agent tasksProvider-reported scoreAdvanced coding-agent challenges

An open benchmark for AI coding agents on real-world React Native implementation tasks, emphasizing working app behavior, recommended architecture choices, and strict constraint adherence.

React Native app implementation tasksFramework-specific app development evaluationProduction mobile app engineering

A coding-agent benchmark for realistic React work, with rubrics that check production concerns such as performance, accessibility, correctness, and code quality.

51 production React tasksPass@1 weighted rubric scoreProduction frontend engineering

An agentic GPU-kernel benchmark that measures how much of the hardware roofline a model's correct, audit-clean kernels reach on six demanding CUDA and Triton problems.

6 GPU-kernel optimization problemsMean peak fraction of hardware roofline over valid cellsAgentic GPU systems engineering

A Vercel benchmark for AI coding agents on Next.js code generation and migration tasks, reporting success rate, average execution time, and an AGENTS.md documentation-assisted split.

24 Next.js code generation and migration tasksAgent task completion with withheld Vitest assertionsFramework-specific web application engineering

A display-only SWE-bench Verified reference from Arcee AI's Trinity-Large-Thinking comparison chart.

Repository task completionAgent scaffold benchmarkProfessional software engineering

A text-to-SQL benchmark over realistic warehouse-scale schemas, reported by Interfaze for model comparison.

Text-to-SQL queriesExecution accuracyEnterprise text-to-SQL

SciCode evaluates language models on generating code for realistic scientific research problems across 16 subfields of physics, math, chemistry, biology, and material science. Problems decompose into 338 subproblems requiring domain knowledge recall, scientific reasoning, and precise code synthesis. Based on real scripts from published research.

80

A display-only Artificial Analysis coding index.

Cross-benchmark coding indexAggregated model scoreDisplay-only external reference

A display-only Artificial Analysis leaderboard for coding-agent systems, combining agent harnesses, host models, and execution settings across software-engineering benchmarks.

Composite over DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnAAverage pass@1 indexReal-world coding-agent workflows

A display-only Artificial Analysis SciCode score.

Scientific coding subproblemsTask success rateScientific programming

A display-only Artificial Analysis coding metric for agentic coding and terminal use on a harder Terminal-Bench slice.

Agentic coding and terminal tasksTask success rateProfessional software engineering

A display-only MiniMax provider benchmark for end-to-end coding-agent and product-building tasks.

End-to-end coding-agent tasksTask success rateFrontier coding-agent workflows

A display-only provider benchmark for generating or manipulating SVG outputs from natural-language requirements.

SVG generation and editing tasksTask success rateVisual coding and structured graphics generation

A display-only benchmark for difficult GPU kernel implementation and optimization tasks.

Hard GPU kernel coding tasksTask success rateSpecialized systems programming

Evaluates coding agents on 333 multimodal game-development tasks in Godot, spanning 2D graphics, 3D graphics, user interfaces, and gameplay logic.

333 tasks from 88 tutorialsPass@1 on the full task set with 95% confidence intervalsMultimodal game development in Godot 4.4.1

A systems and software-engineering benchmark from ByteDance Seed that evaluates agents on long-horizon edge tasks using time-budgeted learning curves rather than a single static pass rate.

Systems and software-engineering tasksTime-budgeted agent learning curvesLong-horizon engineering

ARC Prize fluid-intelligence benchmark using novel visual grid transformations.

Semi-private ARC-AGI-1 evaluation setVerified accuracyAbstract visual reasoning

A dataset for evaluating language models on multistep soft reasoning tasks specified in natural language narratives. Tests the ability to perform complex, structured reasoning.

Multi-step reasoningNarrative-based reasoningComplex reasoning tasks

A suite of 23 challenging tasks from the BIG-Bench collaborative benchmark where prior language models failed to exceed average human performance, even with chain-of-thought prompting.

23 tasksMixed reasoning tasksAdvanced reasoning

A reading-comprehension benchmark requiring discrete reasoning over paragraphs, reported in DeepSeek-V4 base-model evaluations.

Paragraph reasoning questionsF1Reading and numerical reasoning

A commonsense natural-language inference benchmark reported in DeepSeek-V4 base-model evaluations.

Commonsense completion questionsExact matchCommonsense reasoning

A commonsense coreference benchmark reported in DeepSeek-V4 base-model evaluations.

Coreference resolution questionsExact matchCommonsense reasoning

A Chinese Winograd Schema Challenge benchmark reported in DeepSeek-V4 base-model evaluations.

Chinese coreference questionsExact matchChinese commonsense reasoning

A word-chain reasoning benchmark that tests planning, recall, constraint following, and vocabulary depth by asking models to extend non-repeating edit-distance-1 chains.

50 starting words × 3 trialsDifficulty-weighted word-chain reasoningOpen-ended lexical planning

Tests whether model judgments rank argumentative critiques in the same order as expert human ratings across philosophy, AI alignment, and other concept-heavy texts.

224 texts and 608 within-text critique pairsAverage pairwise-ranking loss against expert ratingsFuzzy, expert-rated argumentative reasoning

A multi-step verifiable reasoning benchmark that evaluates whether models can solve pencil puzzles with unique solutions.

300 evaluation puzzlesDirect and agentic puzzle solve rateMulti-step verifiable reasoning

A long-context benchmark that measures whether models can actually use extended context windows for reasoning and retrieval.

Long-context tasksExtended-context retrieval and reasoningHard long-context

A long-context benchmark for memory, retrieval, and multi-round coherence over large contexts.

Long-context retrievalMulti-round long-context evaluationHard long-context

MRCR v2 slice focused on long-context retrieval at 64K-128K lengths.

8-needle retrieval tasksLong-context retrievalLong-context reasoning

MRCR v2 slice focused on very long contexts at 128K-256K lengths.

8-needle retrieval tasksVery-long-context retrievalVery long-context reasoning

MRCR v2 slice focused on retrieval across 256K-512K-token contexts.

100 eight-needle retrieval examplesMean sequence-matcher ratioVery long-context retrieval

MRCR v2 slice focused on retrieval across 512K-1M-token contexts.

100 eight-needle retrieval examplesMean sequence-matcher ratioMillion-token retrieval

Long-context graph traversal benchmark using breadth-first search tasks.

Graph traversal tasksLong-context graph reasoningAlgorithmic long-context reasoning

Long-context benchmark for recovering parent relationships inside graph tasks.

Graph parent-retrieval tasksLong-context graph reasoningAlgorithmic long-context reasoning

A million-token MRCR long-context retrieval benchmark reported in DeepSeek-V4 model evaluations.

Million-token retrievalLong-context retrieval MMRMillion-token long context

A million-token CorpusQA long-context question-answering benchmark reported in DeepSeek-V4 model evaluations.

Million-token corpus question answeringLong-context QA accuracyMillion-token long context

A benchmark measuring fluid intelligence and novel abstract reasoning through visual grid puzzles. Models must identify patterns in input-output pairs and generate the correct output for unseen inputs. Considered the hardest public reasoning benchmark — average individual human performance is 66%.

Visual pattern completion and abstract reasoningGrid transformation puzzles with novel rulesExpert-level — hardest public reasoning benchmark

An interactive successor to ARC-AGI-2 that evaluates whether an AI agent can learn unfamiliar task mechanics through action and feedback.

Interactive game-like tasks with hidden rulesAgentic task completion under a capped evaluation budgetFrontier agentic reasoning

A multistage statistical-reasoning benchmark for genomics and biological-data analysis agents.

129 genomics statistical-analysis workflowsEval-level pass rate across dependent analysis decisionsLong-horizon scientific reasoning

A long-context retrieval benchmark that measures whether a model can recover relevant information embedded deep inside very long contexts.

Long-context retrievalNeedle-in-a-haystack recallLong-context memory

The hardest subset of GPQA featuring the most challenging graduate-level science questions. Sometimes reported separately from the standard GPQA benchmark.

Expert-level science questionsMultiple choice questionsGraduate-level scientific reasoning

A display-only Artificial Analysis long-context reasoning evaluation.

Long-context reasoning tasksAccuracyLong-context reasoning

A display-only Artificial Analysis metric for research-level physics reasoning.

Research-level physics questionsAccuracyResearch-level physics reasoning

A benchmark that tests whether AI models challenge nonsensical, ill-posed, or logically flawed prompts instead of confidently generating incorrect answers. Measures the critical ability to push back on bad input.

Nonsensical and flawed prompts across multiple domainsPrompt challenge and refusal evaluationRobustness and critical reasoning

An automated evaluation framework using 1,000+ real-world user tasks covering reasoning, planning, coding, and creative writing. Highly correlated with Chatbot Arena human preference rankings.

1,024 real-world tasksReal-world task evaluationDiverse real-world scenarios

Professional chart understanding across 100 specialized chart types with expert-set answer tolerances.

100 specialized chart typesAccuracy without toolsProfessional chart reasoning

Professional chart understanding with a container, standard libraries, and image cropping.

100 specialized chart typesAccuracy with toolsProfessional chart reasoning

Generates CadQuery code from multi-view renders and scores geometric similarity by voxel intersection-over-union.

1,000-file Vision2Code subsetVoxel IoUProgrammatic CAD generation

Generates CadQuery code from multi-view renders with image inspection and code-execution tools.

1,000-file Vision2Code subsetVoxel IoU with toolsProgrammatic CAD generation

Professional document understanding over 100 real-world PDFs from ten domains.

100 professional document promptsMean criteria pass rateProfessional document reasoning

Professional document understanding with a container, standard libraries, and image cropping.

100 professional document promptsMean criteria pass rate with toolsProfessional document reasoning

Grounded numerical reasoning over a corpus of historical U.S. Treasury Bulletin documents.

Historical Treasury Bulletin questionsAgentic grounded QA accuracyProfessional document reasoning

A broad multimodal reasoning benchmark spanning charts, diagrams, tables, and academic visual question answering.

Multimodal academic reasoningImage + text question answeringFrontier multimodal

A harder multimodal benchmark for frontier models that combines text with images, diagrams, charts, and academic visual reasoning tasks.

Multimodal academic reasoningImage + text question answeringFrontier multimodal

A display-only Artificial Analysis MMMU-Pro score.

Multimodal academic reasoningImage + text question answeringFrontier multimodal

A native OCR benchmark for reading text from images across multilingual scripts, low-quality scans, handwriting, structured layouts, charts, and screenshots.

Image OCR tasksAccuracyNative visual text understanding

An end-to-end document understanding benchmark over long, layout-rich PDFs with tables, equations, headers, footnotes, and multi-column flows.

Layout-rich PDF understandingMean accuracyComplex document processing

A speech-recognition benchmark on the cleaned Artificial Analysis VoxPopuli subset, reported as word error rate where lower is better.

Speech-to-text transcriptionWord error rateAudio speech recognition

A display-only Design Arena website-generation Elo score surfaced on OpenRouter model benchmark pages.

Website generation comparisonsEloDesign and website generation

A benchmark for grounded reasoning over office-style documents, spreadsheets, charts, and business artifacts.

Document and spreadsheet tasksGrounded QA over office artifactsEnterprise grounded reasoning

A tool-augmented MathVision variant that permits Python during visual mathematics reasoning.

Visual mathematics problems with PythonImage and mathematics reasoning with toolsAdvanced multimodal mathematics

A Python-assisted BabyVision evaluation for fine-grained visual perception and grounded reasoning.

Visual perception tasks with PythonTool-augmented multimodal scoreFine-grained visual perception

A Python-assisted ZeroBench_main evaluation reported as pass@5.

Visual reasoning questions with PythonPass@5Tool-augmented visual reasoning

A forced-answer WorldVQA variant for atomic visual world knowledge.

Atomic visual world-knowledge questionsForced-answer visual QAFine-grained visual knowledge

A document-understanding benchmark for parsing and reasoning over complex document layouts.

Complex document-understanding tasksDocument-understanding scoreGrounded document reasoning

Moonshot AI's internal benchmark for atomic visual perception capabilities.

Internal atomic visual-perception tasksInternal evaluation scoreFine-grained visual perception

Tool-augmented MMMU-Pro variant that allows Python assistance during multimodal reasoning.

Multimodal academic reasoningImage + text question answering with PythonFrontier multimodal

A document understanding benchmark used in frontier-model comparison tables to measure extraction and grounded reasoning quality on complex documents.

Document understanding tasksDocument understanding benchmarkGrounded document reasoning

A display-only Liquid AI extraction metric measuring the share of image-to-JSON outputs that parse as strict JSON.

Image-to-JSON extractionStrict JSON parseability rateStructured visual extraction

A display-only Liquid AI extraction metric measuring field-name agreement between requested schema fields and extracted JSON fields.

Image-to-JSON extractionSchema field F1Structured visual extraction

A display-only Liquid AI extraction metric measuring judged agreement between extracted values and the source image.

Image-to-JSON extractionVLM-judged extraction accuracyStructured visual extraction

A grounded visual QA benchmark focused on answering practical questions about real-world images and scenes.

Real-world visual question answeringImage-grounded QAGeneral visual reasoning

A video understanding benchmark that allows subtitle access when answering multimodal questions about videos.

Video understandingVideo QA with subtitle contextMultimodal video reasoning

A stricter Video-MME setting that removes subtitle help and tests video understanding from visual and audio context alone.

Video understandingVideo QA without subtitle contextMultimodal video reasoning

A comprehensive benchmark for multimodal large language models on video understanding, covering temporal reasoning, perception, and question answering over videos.

Video understandingVideo QA and analysisBroad multimodal video reasoning

A visual mathematics benchmark that tests whether a model can solve math problems grounded in diagrams, equations, figures, and other visual inputs.

Visually grounded math problemsImage + math reasoningAdvanced multimodal mathematics

A multimodal math benchmark for visually grounded mathematical reasoning and answer generation.

Visually grounded math problemsMultimodal mathematical reasoningAdvanced multimodal mathematics

A multimodal benchmark for dynamic mathematical reasoning over visual and structured inputs.

Dynamic visual math problemsMultimodal mathematical reasoningAdvanced multimodal mathematics

A general visual question-answering benchmark used in provider tables for real-image reasoning quality.

Real-image visual QAImage-grounded QAGeneral visual reasoning

A conversational visual QA benchmark that tests multi-turn grounded answering over images and documents.

Conversational visual QAMulti-turn image-grounded QAConversational multimodal reasoning

A long-document multimodal benchmark for grounded reasoning over extended document contexts.

Long document understandingDocument-grounded reasoningLong-context document reasoning

An OCR-focused benchmark for reading and extracting text from visually complex documents and images.

Optical character recognitionText extraction from images and documentsDocument reading

A diagram understanding benchmark focused on scientific and educational visual question answering.

Diagram understandingDiagram-grounded QAStructured visual reasoning

A visual counting benchmark that tests whether a model can count objects and entities reliably in complex scenes.

Visual counting tasksImage-grounded countingFine-grained visual perception

A referring-expression grounding benchmark averaged across RefCOCO variants to test whether a model can localize described objects correctly.

Referring-expression groundingGrounded visual localizationFine-grained visual grounding

A visual detection and grounding benchmark slice used to compare zero-shot object understanding across diverse domains.

Out-of-distribution object understandingDetection and groundingRobust visual grounding

A grounded visual reasoning benchmark focused on evidence-based question answering over real images.

Evidence-based visual QAGrounded image reasoningGrounded multimodal reasoning

A video extension of MMMU-style multimodal reasoning over expert questions grounded in temporal media.

Video-grounded expert reasoningVideo + text reasoningFrontier multimodal video reasoning

A multi-task video understanding benchmark averaged across MLVU categories.

General video understandingVideo QA and understandingBroad multimodal video reasoning

A long-video understanding benchmark for retrieving and reasoning over information distributed across extended video inputs.

Long-form video question answeringLong-video understanding scoreExtended temporal reasoning

A benchmark for evaluating multimodal models on video understanding tasks across multiple disciplines, emphasizing temporal reasoning and comprehension over video content.

Video understandingVideo reasoning benchmarkMulti-disciplinary multimodal video reasoning

A GUI-grounding benchmark for 1,581 instructions in full-screen, high-resolution professional interfaces. It tests where a target is, not whether an agent can finish the surrounding workflow.

1,581 grounding instructionsStatic interface element localizationProfessional GUI grounding

A visual agent benchmark for interface reasoning and task execution over screenshots or software surfaces.

Visual agent and interface reasoningScreenshot-grounded task reasoningComputer-use visual reasoning

An evaluation focused on professional domain expertise and task delivery quality in office-style knowledge work.

Professional office deliveryELO-style office benchmarkProfessional knowledge work

A multimodal medical multiple-choice benchmark covering clinical images such as X-rays, histology, and dermatology.

2,000 multimodal medical questionsMedical visual MCQClinical multimodal reasoning

A multi-step visual reasoning benchmark with pass@5 reporting and optional tool use.

100 visual reasoning questionsMulti-step visual reasoningTool-augmented visual reasoning

A multimodal coding benchmark for turning visual designs into working frontend implementations.

Design-to-code tasksVisual input to frontend implementationMultimodal coding

A vision-language coding benchmark for generating correct code from visual and multimodal inputs.

Multimodal coding tasksVision-language code generationMultimodal coding

A benchmark for converting visual references into functional web implementations.

Screenshot-to-web tasksVisual reference to web implementationMultimodal web generation

A multimodal retrieval and extraction benchmark over image-heavy task settings.

Visual retrieval tasksImage-grounded retrieval and extractionMultimodal retrieval

A multimodal search benchmark for retrieval and grounded answering across mixed-media inputs.

Multimodal search tasksMixed-media retrieval and grounded answeringMultimodal search

A harder MMSearch variant for multimodal retrieval and grounded tool-use workflows.

Hard multimodal search tasksAdvanced mixed-media retrieval benchmarkAdvanced multimodal search

A visual question answering benchmark focused on straightforward image-grounded understanding.

Visual QA tasksImage-grounded question answeringGeneral visual understanding

A grounded multimodal factuality benchmark for evidence-linked answer correctness.

Grounded factuality tasksEvidence-linked multimodal factualityGrounded multimodal factuality

A vision-centric benchmark for high-level multimodal reasoning and perception quality.

Frontier multimodal reasoning tasksVision-centric reasoning benchmarkFrontier multimodal

A scientific chart reasoning benchmark that tests whether models can understand, interpret, and reason about complex scientific visualizations including plots, diagrams, and data charts.

Scientific chart reasoningChart understanding and reasoningScientific visualization reasoning

Tool-free variant of CharXiv that isolates raw visual reasoning ability without code execution or tool augmentation.

Scientific chart reasoning (tool-free)Chart understanding without toolsScientific visualization reasoning

A multimodal benchmark for fine-grained visual perception and grounded reasoning tasks.

Visual perception tasksMultimodal visual reasoningFine-grained visual perception

A multimodal variant of SWE-bench that adds visual context (screenshots, design mockups) to software engineering issue descriptions, testing whether models can leverage visual information for code generation.

Multimodal software engineering tasksCode patch generation with visual contextFrontier multimodal coding

An agentic spatial reasoning benchmark reported as a normalized score.

Spatial reasoning from blueprintsNormalized scoreAgentic spatial reasoning

Raw score on realistic multi-turn healthcare conversations graded against expert-written rubrics.

5,000 multi-turn patient conversationsRaw rubric scoreRealistic healthcare conversations

HealthBench score after applying a verbosity penalty to model responses.

5,000 multi-turn patient conversationsLength-adjusted rubric scoreRealistic healthcare conversations

Raw score on physician-authored clinical consult, documentation, and research conversations.

525 physician-authored conversationsRaw rubric scoreProfessional clinical tasks

Computational biology challenges that independent human experts were able to solve.

Human-solvable computational biology investigationsTask scoreExpert computational biology

Computational biology challenges with objective answers that remained unsolved by independent human experts.

Human-difficult computational biology investigationsTask scoreFrontier computational biology

Analysis of spatial transcriptomics data across externally validated biological problems.

115 externally validated spatial transcriptomics problemsTask scoreProfessional bioinformatics

Single-cell RNA sequencing analysis tasks spanning common bioinformatics workflows.

195 single-cell RNA sequencing problemsTask scoreProfessional bioinformatics

Predicts mutation effects by ranking mutant protein sequences against wild type and comparing against laboratory measurements.

Hard protein mutation-effect ranking tasksRank correlationComputational protein science

Generates novel protein sequences under family, topology, globularity, and structural-motif constraints.

Constrained protein-sequence design tasksComposite scoreComputational protein design

Chemistry tasks covering spectroscopy, synthesis planning, reaction prediction, and chemical structure images.

Organic chemistry reasoning tasksTask scoreExpert organic chemistry

Detects and fixes errors in molecular-biology protocols using document, code, and web-search tools.

Molecular-biology protocol troubleshootingTask scoreExpert laboratory protocols

Extends online molecular-biology protocols in additional directions using document and web-search tools.

Molecular-biology protocol extensionTask scoreExpert laboratory protocols

A display-only Artificial Analysis model-openness index.

Model openness assessmentIndex scoreDisplay-only external reference

An independently evaluated MMLU-Pro result from Artificial Analysis.

Professional multi-subject questionsAccuracyProfessional knowledge and reasoning

A comprehensive multiple-choice question answering test covering 57 tasks including elementary mathematics, US history, computer science, law, and more. Tests knowledge across diverse academic subjects from high school to professional level.

57 subjectsMultiple choice questionsElementary to professional level

A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Designed to be difficult even for skilled non-experts with access to Google.

448 questionsMultiple choice questionsGraduate level

A display-only GPQA Diamond reference from provider comparison charts.

Graduate-level science questionsMultiple choice questionsGraduate level

An expanded version of GPQA that evaluates graduate-level knowledge and reasoning capabilities across 285 disciplines, providing comprehensive coverage of academic domains.

285 disciplinesMultiple choice questionsGraduate level

An enhanced version of MMLU with 10 answer choices instead of 4, featuring more reasoning-focused questions that better differentiate frontier models.

Multiple subjects10-way multiple choiceProfessional level

A human-centric exam benchmark for general knowledge and reasoning reported in DeepSeek-V4 base-model evaluations.

General academic and professional exam questionsExact matchGeneral knowledge

An expert-authored benchmark designed to probe frontier knowledge and reasoning. BenchLM keeps protocol differences visible because tool-assisted and closed-book HLE runs answer different questions.

Expert-level questionsOpen-ended and multiple choiceFrontier expert level

A verified and revised Humanity's Last Exam set that removes uncertain items and repairs fixable questions before evaluating frontier expert reasoning.

1,811 verified or revised expert questionsFull-set accuracyFrontier multidisciplinary expert reasoning

A benchmark of realistic biology-research tasks involving literature, figures, tables, databases, and bioinformatics files.

Nearly 1,900 biology-research tasksAggregate accuracyReal-world biology research

A benchmark for research-level scientific reasoning, designed to separate frontier models on difficult science tasks that mix domain knowledge with deep reasoning.

Research-level science tasksScientific reasoning benchmarkResearch frontier

A display-only intelligence index published by Artificial Analysis that aggregates provider-reported and benchmark-derived signals into a single model-level score.

Cross-benchmark intelligence indexAggregated model scoreDisplay-only external reference

A display-only Artificial Analysis GPQA Diamond score.

Graduate-level science questionsAccuracyGraduate-level science reasoning

A display-only Artificial Analysis Humanity's Last Exam score.

Expert-level questionsAccuracyFrontier expert reasoning

A display-only Artificial Analysis factual knowledge index.

Knowledge questionsIndex scoreBroad factual knowledge

A display-only Artificial Analysis knowledge metric for the proportion of correctly answered questions.

Knowledge questionsAccuracyBroad knowledge

A display-only Artificial Analysis factuality metric for the rate of incorrect answers among non-correct responses.

Knowledge questionsHallucination rateFactuality

A benchmark that evaluates the ability of language models to answer short, fact-seeking questions accurately. Focuses on factual correctness rather than reasoning complexity.

Factual questionsShort-form Q&AFactual accuracy focused

A Chinese short-form factuality benchmark reported by DeepSeek for V4 model evaluations.

Chinese factual questionsShort-form factual QAFactual accuracy focused

A science question-answering benchmark that tests whether models can apply a small open-book set of elementary science facts to multi-step reasoning questions.

Elementary science questions4-way multiple choiceElementary science reasoning

A harder subset of OpenAI's HealthBench for evaluating open-ended medical and health reasoning with rubric-based grading.

1,000 health promptsOpen-ended health evaluationAdvanced health reasoning

An open benchmark for clinician-facing model responses across care consult, writing and documentation, and medical research tasks.

Clinician chat tasksRubric-graded open-ended responsesProfessional clinical workflows

A medical multiple-choice benchmark spanning many specialties with 10 answer options per question.

2,450 medical multiple-choice questionsMedical MCQProfessional medical knowledge

A research-focused FrontierScience evaluation variant for scientific investigation and problem solving.

Scientific research problemsResearch evaluationFrontier scientific research

A benchmark designed to measure whether language models produce truthful answers instead of repeating common misconceptions or misleading falsehoods.

Truthfulness and misconception resistanceQuestion answeringHallucination and factuality stress test

Tool-free variant of Humanity's Last Exam that isolates a model's raw frontier reasoning.

Expert-level questionsTool-free expert QAFrontier expert level

A display-only MMLU-Pro reference from Arcee AI's Trinity-Large-Thinking launch chart.

Professional academic QA10-way multiple choiceProfessional level

A harder refresh of MMLU intended to keep broad knowledge evaluation useful after the original benchmark became too easy for frontier models.

Broad academic QAMultiple choice questionsAdvanced general knowledge

A multilingual MMLU-style benchmark reported in provider evaluation tables.

Multilingual academic QAExact matchBroad multilingual knowledge

A Chinese-language academic and professional benchmark spanning humanities, social science, STEM, and applied subjects.

Chinese academic and professional examsMultiple choice questionsHigh school to professional level

A Chinese multitask academic benchmark reported in DeepSeek-V4 base-model evaluations.

Chinese academic QAExact matchBroad Chinese knowledge

A multilingual/localized knowledge benchmark reported in DeepSeek-V4 base-model evaluations.

Localized multilingual knowledge questionsExact matchMultilingual knowledge

A parametric factuality benchmark reported in DeepSeek-V4 base-model evaluations.

Parametric factual recallExact matchFactual accuracy focused

A reading and trivia question-answering benchmark reported in DeepSeek-V4 base-model evaluations.

Trivia and reading-comprehension QAExact matchGeneral factual QA

An AfterQuery benchmark of open-ended financial analysis that requires models to read financial data, make assumptions, and return exact answers.

Professional financial-analysis questionsOpen-ended financial QA with exact-match gradingProfessional finance reasoning

MMLU-style knowledge evaluation across 42 high- and low-resource languages.

Knowledge questions across 42 languagesAverage accuracyMultilingual knowledge

Culturally grounded knowledge comprehension across ten Indic languages and English.

Knowledge tasks across 11 languagesAverage accuracyMultilingual Indic knowledge

An independently evaluated multilingual knowledge result from Artificial Analysis.

Multilingual knowledge questionsAccuracyMultilingual professional knowledge

A multilingual benchmark that translates 250 grade school math problems from GSM8K into 10 typologically diverse languages: Bengali, German, Spanish, French, Japanese, Russian, Swahili, Telugu, Thai, and Chinese.

250 problems × 11 languagesMath word problemsGrade school math, multilingual

A multilingual extension of professional-level academic evaluation across many languages.

Multilingual professional QAMultilingual multiple choiceProfessional multilingual

A broad multilingual benchmark row from Qwen's launch comparisons intended to measure cross-lingual capability beyond a single language family.

Broad multilingual evaluationCross-lingual benchmarkBroad multilingual capability

A multilingual benchmark used in provider tables to measure inclusive language coverage and cross-lingual understanding beyond common high-resource languages.

Cross-lingual understandingMultilingual benchmarkBroad multilingual capability

A multilingual mathematical reasoning benchmark that tests whether math performance transfers across languages rather than only in English.

Multilingual math problemsCross-lingual mathematical reasoningAdvanced multilingual reasoning

A lighter multilingual benchmark slice published in provider tables for broad cross-lingual transfer and understanding.

Multilingual transfer tasksCross-lingual benchmarkBroad multilingual capability

A multilingual instruction-following and understanding benchmark row published in Qwen's launch comparisons.

Multilingual instruction followingCross-lingual benchmarkAdvanced multilingual instruction following

A multilingual extension of SWE-bench covering 300 problems across 9 programming languages, testing code generation and bug fixing beyond Python.

300 problems across 9 languagesMulti-language code patch generationProfessional multilingual software engineering

A display-only multilingual retrieval benchmark reported by Liquid AI for LFM2.5 retriever models, using NDCG@10 across 11 languages.

Multilingual document retrievalNDCG@10 averageMultilingual retrieval

A display-only multilingual QA retrieval benchmark reported by Liquid AI for LFM2.5 retriever models, using Recall@20 across 11 languages.

Cross-lingual open-domain QA retrievalRecall@20 averageMultilingual retrieval

A benchmark of 541 prompts built from 25 verifiable instruction types. It tests whether a model follows checkable constraints such as keyword, length, casing, and response-format requirements.

541 prompts across 25 instruction typesConstrained generationInstruction precision

IFBench evaluates precise instruction-following generalization on 58 challenging, verifiable out-of-domain constraints. Unlike IFEval which tests familiar constraint types, IFBench specifically measures how well models follow novel instructions they haven't been optimized for, exposing overfitting to common instruction patterns.

58

A display-only Artificial Analysis IFBench score.

Verifiable instruction constraintsConstraint satisfaction accuracyInstruction precision

A structured-output benchmark from Interfaze measuring whether extracted JSON leaf values exactly match verified ground truth.

Structured output extractionValue accuracyProduction structured-output reliability

Proof-based olympiad performance on all six IMO 2026 problems.

6 proof-based problemsOfficial-style proof scoreInternational olympiad mathematics

Research-level mathematics problems with unique programmatically verified closed-form answers.

25 private research-level mathematics problemsAccuracy without toolsResearch mathematics

Research-level mathematics problems with unique programmatically verified closed-form answers and tool access.

25 private research-level mathematics problemsAccuracy with toolsResearch mathematics

Final-answer research mathematics problems drawn from recent arXiv abstracts.

49 recent research-mathematics problemsFinal-answer accuracyResearch mathematics

Final-answer research mathematics problems drawn from recent arXiv abstracts with tool access.

49 recent research-mathematics problemsFinal-answer accuracy with toolsResearch mathematics

An independently evaluated AIME 2025 result from Artificial Analysis.

30 AIME 2025 problemsAccuracyOlympiad mathematics

An independently evaluated MATH-500 result from Artificial Analysis.

500 competition mathematics problemsAccuracyHigh school to undergraduate mathematics

A 15-question, 3-hour examination where each answer is an integer from 000 to 999. Serves as the intermediate step between AMC 10/12 and the USA Mathematical Olympiad (USAMO).

15 problemsInteger answers 000-999High school olympiad level

The 2024 edition of AIME, maintaining the same format of 15 challenging mathematics problems with integer answers from 000 to 999.

15 problemsInteger answers 000-999High school olympiad level

The most recent AIME examination, featuring 15 challenging mathematics problems testing olympiad-level mathematical reasoning with integer answers from 000-999.

15 problemsInteger answers 000-999High school olympiad level

A grade-school mathematical reasoning benchmark reported in DeepSeek-V4 base-model evaluations.

Grade-school math word problemsExact matchGrade-school math

A competition-style mathematical reasoning benchmark reported in DeepSeek-V4 base-model evaluations.

Competition math problemsExact matchAdvanced math reasoning

A Chinese mathematical reasoning benchmark reported in DeepSeek-V4 base-model evaluations.

Chinese math problemsExact matchMath reasoning

A display-only AIME25 reference from Arcee AI's Trinity-Large-Thinking launch chart.

15 problemsInteger answers 000-999High school olympiad level

A prestigious high school mathematics competition hosted jointly by Harvard and MIT, featuring challenging problems across various mathematical disciplines.

Tournament problemsCompetition mathematicsHigh school olympiad level

The 2024 February edition of the Harvard-MIT Mathematics Tournament, continuing the tradition of challenging high school mathematics competition.

Tournament problemsCompetition mathematicsHigh school olympiad level

The most recent February edition of the Harvard-MIT Mathematics Tournament, featuring the latest challenging problems in competitive mathematics.

Tournament problemsCompetition mathematicsHigh school olympiad level

A challenging mathematical olympiad competition featuring problems that test advanced mathematical reasoning and problem-solving skills at the olympiad level.

Olympiad problemsMathematical olympiadMathematical olympiad level

A curated subset of 500 problems from the MATH dataset, covering algebra, counting and probability, geometry, intermediate algebra, number theory, prealgebra, and precalculus.

500 problemsFree-form mathematical answersHigh school to undergraduate

A 2026 American Invitational Mathematics Examination snapshot used in frontier-model comparison tables for mathematical reasoning.

Competition math problemsShort-answer mathematicsOlympiad-style mathematics

The three official theory problems from the 2025 International Physics Olympiad, scored with blinded human evaluation.

3 olympiad theory problemsPhysics olympiad theoryInternational olympiad physics

A February 2025 HMMT slice used in exact-value provider tables for advanced contest-math reasoning.

Competition math problemsContest mathematicsOlympiad-style mathematics

A November 2025 HMMT slice for high-end mathematical reasoning comparisons.

Competition math problemsContest mathematicsOlympiad-style mathematics

A February 2026 HMMT slice used in newer frontier-model math comparisons.

Competition math problemsContest mathematicsOlympiad-style mathematics

A challenging mathematical reasoning benchmark reported in DeepSeek-V4 model evaluations.

Advanced mathematical answer generationPass@1 math benchmarkOlympiad-level mathematics

A high-difficulty mathematical reasoning benchmark reported in DeepSeek-V4 model evaluations.

Advanced mathematical reasoningPass@1 math benchmarkFrontier math reasoning

A shortlist subset of the Apex mathematical reasoning benchmark reported in DeepSeek-V4 model evaluations.

Advanced mathematical reasoningPass@1 math benchmarkFrontier math reasoning

A multimodal mathematical reasoning benchmark that tests whether models can answer visually grounded math questions correctly.

Multimodal math questionsVisual and structured mathematical QAAdvanced mathematical reasoning

Legacy FrontierMath values retained for historical model pages. This field is not used in current rankings because it can mix prior benchmark versions and slices.

Historical aggregateOpen-ended mathematical reasoning with tool accessResearch-level mathematics

Epoch AI's corrected v2 core FrontierMath suite of private advanced mathematics problems. Models can reason iteratively and use Python; scores are pass rates on the private set.

295 private advanced mathematics problemsPython-enabled iterative mathematical problem solvingFrom olympiad-plus to early research mathematics

Epoch AI's corrected v2 Tier 4 expansion, a separate set of exceptionally difficult research-level mathematics problems evaluated with Python-enabled iterative reasoning.

43 private extreme-difficulty mathematics problemsPython-enabled iterative mathematical problem solvingResearch-level mathematics requiring hours or days of expert work

The premier US mathematical olympiad competition, featuring proof-based problems that require deep mathematical insight and rigorous argumentation at the highest competition level.

6 proof-based problemsMathematical proof constructionInternational olympiad level

Evaluates Korean expert-level knowledge across 45 subjects. 20% of questions require Korean cultural context.

35,030 questionsMultiple choice questionsElementary to professional level in Korean

A filtered hard subset of KMMLU containing ~5,000 questions that most models get wrong.

~5,000 questionsMultiple choice questionsAdvanced Korean reasoning

Cleaned KMMLU from national technical qualification exams, with errors removed, decontaminated, and deduplicated.

~3,500 questionsTechnical multiple choiceIndustrial/technical

Korean National Professional Licensure exams evaluating professional-grade knowledge.

~2,500 questionsProfessional licensure examsProfessional

Evaluates Korean culture and linguistics.

1,995 questionsCultural/linguistic QAKorean cultural nuances

Evaluates advanced Korean linguistic competence.

Linguistics questionsAdvanced linguisticsAdvanced linguistic phenomena

The Korean SAT exam.

Multi-subject examStandardized testHigh school to college level

Korean mathematical reasoning (high-school to Olympiad level).

8,011 instancesMath word problemsOlympiad level

A behavioral benchmark that tests psychological safety across sixteen adversarial multi-turn conversations covering emotional safety, identity, sycophancy, and value integrity.

16 multi-turn scenarios, 72 criteriaJudge-scored behavioral audit with human reviewAdversarial psychological-safety evaluation

A frequently refreshed benchmark with objective scoring across reasoning, coding, agentic coding, mathematics, data analysis, language, and instruction following.

23 objective tasks across 7 categoriesMean of category averagesBroad frontier-model evaluation

Vals AI composite benchmark across professional finance, coding, modeling, and legal-work tasks, including Finance Agent v2, EMB, Terminal-Bench 2.1, Vibe Code Bench, Code Migration, Legal Research, and HLAB.

Finance, coding, spreadsheet modeling, code migration, and legal-work componentsComposite scorePrivate economic-work benchmark composite

A Vals AI comparison of native provider search and Exa across finance-analysis and legal-research tasks.

Finance Agent Benchmark v2 and Legal Research Benchmark tasksAccuracy by model and search-tool combinationProfessional web research with controlled search-tool variants

A Vals AI agent benchmark that gives each system five days to build and run a space program in Kerbal Space Program.

30 progressively harder Kerbal Space Program missionsMission-ladder progress with partial creditLong-horizon autonomous computer use and planning

Vals AI multimodal composite across finance, coding, education, and mortgage-tax task families.

Finance, coding, education, and mortgage-tax componentsComposite scorePrivate multimodal economic-work benchmark composite

An archived Vals benchmark covering retrieval, numerical reasoning, financial modeling, market analysis, earnings, trends, and adjustments.

11 financial analyst task viewsAccuracy with task-level breakdownsProfessional financial analysis

A contamination-resistant agent benchmark for reverse engineering real-world binaries.

Real-world binary reverse-engineering tasksFully solved rate and capability scoreAgentic reverse engineering

A Vals-hosted view of Terminal-Bench 1.0 with easy, medium, and hard task splits.

Terminal tasks split by easy, medium, and hard difficultyAccuracy scoreTerminal-agent execution

Vals AI private benchmark for understanding long-context credit agreements.

Credit-agreement understanding tasksAccuracy scoreProfessional finance document reasoning

Vals AI healthcare benchmark for whether models can support the medical billing process.

Medical billing support tasksAccuracy scoreProfessional healthcare administration

Vals AI healthcare benchmark for whether models can support doctors with administrative work.

Medical administrative support tasksAccuracy scoreProfessional healthcare administration

Vals AI benchmark for mortgage and tax document reasoning, including semantic and numerical extraction task views.

Mortgage and tax extraction tasksAccuracy scoreProfessional mortgage-tax document reasoning

Vals AI automated theorem-proving benchmark.

Automated theorem provingAccuracy scoreFormal proof reasoning

Vals AI legal benchmark with issue, rule, conclusion, interpretation, and rhetoric task views.

Legal reasoning task viewsAccuracy scoreProfessional legal reasoning

Vals AI private question-answer benchmark over Canadian court cases.

Canadian case-law question answeringAccuracy scoreProfessional legal retrieval and reasoning

A long-horizon software engineering benchmark from Datacurve for measuring frontier coding agents on original tasks drawn from active open-source repositories.

113 software engineering tasks across 91 repositories and 5 languagesPass@1 with confidence interval, cost, time, and token metadataLong-horizon software engineering

A long-horizon software engineering benchmark from Abundant AI with multi-hour tasks spanning library reproductions, full-stack product clones, and ML engineering.

20 multi-hour software engineering tasksTask resolution and trajectory reviewUltra-long-horizon software engineering

A cybersecurity benchmark for evaluating LLM agents on full-control V8 exploit synthesis using 16 measured exploit capability flags.

V8 exploit synthesis runsCapability coverage percentage over 16 flagsBrowser exploitation and cybersecurity

Number of advanced cyber-range challenges solved in the joint NIST CAISI and UK AISI preliminary evaluation.

41 advanced cyber-range challengesChallenges solvedAdvanced cyber operations

Average step reached on a 32-step long-horizon cyber range.

32-step long-horizon cyber rangeAverage step reachedLong-horizon cyber operations

Share of runs that completed the 32-step cyber range within the 100-million-token limit.

10 long-horizon runsCompletion rateLong-horizon cyber operations

Share of approved advanced-cyber requests completed rather than refused under standard GPT-5.6 Sol safeguards.

Internal advanced-cyber request setCompletion rateCyber access and safeguards

Share of approved advanced-cyber requests completed rather than refused by GPT-5.6 Sol under Daybreak Blue safeguards.

Internal advanced-cyber request setCompletion rateCyber access and safeguards

Share of approved advanced-cyber requests completed rather than refused by GPT-5.6 Cyber under Daybreak Red access.

Internal advanced-cyber request setCompletion rateCyber access and safeguards

Cybersecurity benchmark for agentic vulnerability analysis and exploit-oriented security tasks.

Security engineering tasksSuccess rateAdvanced cybersecurity

Independent evaluation of AI agents against vulnerable real-world systems in dynamic environments.

197 dynamic cyber tasksTasks solvedEasy through elite cyber operations

Average success rate across realistic, long-horizon cybersecurity scenarios.

11 cyber scenariosAverage success rateLong-horizon cybersecurity