Tau3-Banking — Mercor evaluation (Tau3-Banking (Mercor))

We show this table for reference; we do not rank on it.

Handling simulated banking-support conversations with tools. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Evaluation results published by Mercor. Published evaluation aggregates only; task contents and dataset license grants are not included.

Pass@1 on Tau3-Banking (Mercor) — October 15, 2026 capture

We mirror the published pass@1 view for Tau3-Banking (Mercor). Opus 5.5 leads the public snapshot at 55.00%, followed by Fable 5.1 (54.00%) and Sonnet 5.5 (54.00%). We do not use these results to rank models overall.

Opus 5.5

Anthropic

Tau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoning

Published margin ±8.2 · 291 reported samples

55.00%

Fable 5.1

Anthropic

Tau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoning

Published margin ±8.9 · 291 reported samples

54.00%

Sonnet 5.5

Anthropic

Tau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoning

Published margin ±8.9 · 291 reported samples

54.00%

26 configurationsAgenticDated Mercor configurationsDisplay onlyUpdated October 15, 2026 capture

Pass@1 table (26 configurations)

Pass@1 results for Tau3-Banking (Mercor)
Rank	Model / configuration	Score	Parameters (B)	Open / closed
1	Opus 5.5AnthropicTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±8.2 · 291 reported samples	55.00%	Not reported	Unknown
2	Fable 5.1AnthropicTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±8.9 · 291 reported samples	54.00%	Not reported	Unknown
3	Sonnet 5.5AnthropicTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±8.9 · 291 reported samples	54.00%	Not reported	Unknown
4	Muse Spark 1.3MetaTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · xhigh reasoningPublished margin ±8.6 · 291 reported samples	52.20%	Not reported	Unknown
5	Gemini 3.8 FlashGoogleTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · high reasoningPublished margin ±8.8 · 291 reported samples	48.80%	Not reported	Unknown
6	GPT-6 AstraOpenAITau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±9.1 · 291 reported samples	46.70%	Not reported	Unknown
7	Qwen3.8-MaxAlibabaTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · xhigh reasoningPublished margin ±8.4 · 291 reported samples	46.70%	Not reported	Unknown
8	Grok 4.6xAITau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · xhigh reasoningPublished margin ±8.2 · 291 reported samples	45.70%	Not reported	Unknown
9	Opus 5AnthropicTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±8.1 · 291 reported samples	44.00%	Not reported	Unknown
10	GPT-6 SolOpenAITau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±8.6 · 291 reported samples	43.00%	Not reported	Unknown
11	GPT-5.6 Sol (Pro)OpenAITau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±8.2 · 291 reported samples	43.00%	Not reported	Unknown
12	GPT-6.1 SolOpenAITau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±8.9 · 291 reported samples	43.00%	Not reported	Unknown
13	Haiku 5.5AnthropicTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±8.6 · 291 reported samples	43.00%	Not reported	Unknown
14	GLM-5.3ZhipuTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±8.1 · 291 reported samples	40.50%	Not reported	Unknown
15	GPT-6 LunaOpenAITau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±8.2 · 291 reported samples	40.20%	Not reported	Unknown
16	DeepSeek-V4-Pro-0813DeepSeekTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±8.1 · 291 reported samples	40.20%	Not reported	Unknown
17	GPT-5.6 TerraOpenAITau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±8.1 · 291 reported samples	38.80%	Not reported	Unknown
18	GLM-5.3-FlashZhipuTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±7.6 · 291 reported samples	37.80%	Not reported	Unknown
19	DeepSeek-V4.1-FlashDeepSeekTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±8.4 · 291 reported samples	37.80%	Not reported	Unknown
20	GPT-5.6 LunaOpenAITau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±7.9 · 291 reported samples	35.70%	Not reported	Unknown
21	Kimi K3KimiTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · max reasoningPublished margin ±7.6 · 291 reported samples	34.70%	Not reported	Unknown
22	Grok 4.7xAITau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · xhigh reasoningPublished margin ±7.6 · 291 reported samples	32.60%	Not reported	Unknown
23	MiniMax-M3MiniMaxTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · high reasoningPublished margin ±6.2 · 291 reported samples	21.00%	Not reported	Unknown
24	Gemini 3.5 Flash LiteGoogleTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · high reasoningPublished margin ±6 · 291 reported samples	18.60%	Not reported	Unknown
25	Nemotron 3 UltraNVIDIATau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · high reasoningPublished margin ±6 · 291 reported samples	18.60%	Not reported	Unknown
26	InklingThinking MachinesTau half-duplex agent (200 max steps, 10 tool error limit, 3 hour time limit) · high reasoningPublished margin ±5.8 · 291 reported samples	15.50%	Not reported	Unknown

How to read this Mercor evaluation

We captured Mercor's Tau3-Banking results on October 15, 2026. The table preserves 26 published configurations, their source identifiers, reasoning settings, and disclosed pass@1 values. These are results from Mercor's evaluation setup; we did not rerun them.

This dated source table is display only and does not enter overall or category rankings. Source configurations retain their external identity, including Pro descriptors and different effort settings.

The headline is the source page's selected Pass@1 metric. The task set, harness, tools, and grader belong to this Mercor run and may differ from another evaluator's results.

The row labels retain published error margins and sample counts when supplied. Missing margins and counts remain unreported; a sample count is not used to reconstruct a task denominator. Except where the dedicated source defines an interval, the confidence level is unreported. Equal displayed scores and overlapping margins do not establish statistical ties. Capture time does not establish individual evaluation dates.

The general methodology lists programmatic database-state checks and agent action checks; one llm-judged natural-language assertion and a judge of Gemini 3.5 Flash. Its run-count label is k = 3. The linked result page takes precedence where its setup differs, as noted above.

Snapshot

Pass@126 configurationsMercor evaluationDisplay only

Mercor Tau3-Banking results Mercor APEX benchmark index Mercor methodology Source paper Upstream evaluation code

The published Tau3-Banking (Mercor) snapshot places Opus 5.5 first at 55.00%. The third row is 1.00 points behind. The broader top-10 range is 12.00 points, so the table still separates the published systems.

26 configurations are shown for Tau3-Banking (Mercor). The benchmark falls in the Agentic category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. Tau3-Banking (Mercor) is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About Tau3-Banking (Mercor)

Tasks

97 source-reported tasks

Format

Pass@1

Difficulty

Source-specific evaluation

The selected headline is Pass@1. Source model identifiers, effort settings, published margins, and reported sample counts remain attached to the source configuration. Public-set results and Mercor task extensions occupy separate tables.

Tau3-Banking evaluation results Public benchmark source

Freshness and provenance

Version

Tau3-Banking; Mercor Pass@1

Refresh cadence

Reviewed dated source capture

Staleness state

Dated Mercor configurations

Question availability

Upstream task access and terms; only aggregate results mirrored

Dated Mercor configurationsDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does Tau3-Banking (Mercor) measure?

Handling simulated banking-support conversations with tools. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Which model leads the published Tau3-Banking (Mercor) snapshot?

Opus 5.5 currently leads the published Tau3-Banking (Mercor) snapshot with 55.00% pass@1. BenchLM shows this benchmark for display only and does not use it in overall rankings.

How many models are evaluated on Tau3-Banking (Mercor)?

The October 15, 2026 capture snapshot contains 26 source configurations.

Last updated: October 15, 2026 capture · mirrored from the public benchmark leaderboard

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 5,600+ readers.

One email each week. Unsubscribe anytime.