# RedlineBench — Mercor evaluation (RedlineBench (Mercor))

> Negotiating and revising contracts across multiple turns. This table shows Mercor-run configurations for reference and is excluded from model rankings.

Canonical page: https://benchlm.ai/benchmarks/mercorredlinebench

- Category: [Agentic](/agentic)
- Last updated: October 15, 2026 capture

Evaluation results published by [Mercor](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/). Published evaluation aggregates only; task contents and dataset license grants are not included.

We captured Mercor's RedlineBench results on October 15, 2026. The table preserves 12 published configurations, their source identifiers, reasoning settings, and disclosed turn-weighted mean score values. These are results from Mercor's evaluation setup; we did not rerun them.

This dated source table is display only and does not enter overall or category rankings. Source configurations retain their external identity, including Pro descriptors and different effort settings.

The score is a turn-weighted mean, not task pass rate. Mercor uses a uniform Claude Code Harbor setup and a three-model judge panel; sample-task grades come from one judge and are not the leaderboard score. The panel replaces the original Gemini 3.1 judge with Gemini 3.5 Flash.

The row labels retain published error margins and sample counts when supplied. Missing margins and counts remain unreported; a sample count is not used to reconstruct a task denominator. Except where the dedicated source defines an interval, the confidence level is unreported. Equal displayed scores and overlapping margins do not establish statistical ties. Capture time does not establish individual evaluation dates.

The general methodology lists expert-authored rubrics, scored criterion by criterion and a judge of 3-LLM judge panel over expert rubrics. Its run-count label is k = 3. The linked result page takes precedence where its setup differs, as noted above.

- [Mercor RedlineBench results](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/)
- [Mercor APEX benchmark index](https://www.mercor.com/apex/)
- [Mercor methodology](https://www.mercor.com/apex/methodology/)
- [Source paper](https://intelligence.crosby.ai/benchmark/)
- [Upstream evaluation code](https://github.com/crosbylegal/redline-bench)
- [Upstream dataset](https://huggingface.co/datasets/crosbylegal/RedlineBench/tree/main/tasks)

## About RedlineBench (Mercor)

- Tasks: 140 source-reported tasks
- Format: Turn-Weighted Mean Score
- Difficulty: Source-specific evaluation
- Paper: [RedlineBench evaluation results](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/)

The selected headline is Turn-Weighted Mean Score. Source model identifiers, effort settings, published margins, and reported sample counts remain attached to the source configuration. Public-set results and Mercor task extensions occupy separate tables.

RedlineBench (Mercor) is currently displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.

## Leaderboard (12 configurations)

| Rank | Model | Configuration | Creator | Score |
|------|-------|---------------|---------|-------|
| 1 | [GPT-5.6 Sol](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/) | RedlineBench Harbor — Claude Code (gen-only, disable_verif+capture) · max reasoning · Margin not reported · 420 reported samples | OpenAI | 46.80% |
| 2 | [GPT-5.5](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/) | RedlineBench Harbor — Claude Code (gen-only, disable_verif+capture) · xhigh reasoning · Margin not reported · 420 reported samples | OpenAI | 44.10% |
| 3 | [GPT-5.6 Terra](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/) | RedlineBench Harbor — Claude Code (gen-only, disable_verif+capture) · max reasoning · Margin not reported · 420 reported samples | OpenAI | 44.00% |
| 4 | [Opus 5](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/) | RedlineBench Harbor — Claude Code (gen-only, disable_verif+capture) · max reasoning · Margin not reported · 420 reported samples | Anthropic | 44.00% |
| 5 | [GPT-5.6 Luna](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/) | RedlineBench Harbor — Claude Code (gen-only, disable_verif+capture) · max reasoning · Margin not reported · 420 reported samples | OpenAI | 43.30% |
| 6 | [Fable 5](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/) | RedlineBench Harbor — Claude Code (gen-only, disable_verif+capture) · max reasoning · Margin not reported · 420 reported samples | Anthropic | 42.40% |
| 7 | [Gemini 3.6 Flash](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/) | RedlineBench Harbor — Claude Code (gen-only, disable_verif+capture) · high reasoning · Margin not reported · 420 reported samples | Google | 41.60% |
| 8 | [GLM-5.2](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/) | RedlineBench Harbor — Claude Code (gen-only, disable_verif+capture) · max reasoning · Margin not reported · 420 reported samples | Zhipu | 39.60% |
| 9 | [Haiku 5.5](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/) | RedlineBench Harbor — Claude Code (gen-only, disable_verif+capture) · max reasoning · Margin not reported · 420 reported samples | Anthropic | 39.60% |
| 10 | [Kimi K3](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/) | RedlineBench Harbor — Claude Code (gen-only, disable_verif+capture) · max reasoning · Margin not reported · 420 reported samples | Kimi | 39.40% |
| 11 | [Gemini 3.5 Flash](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/) | RedlineBench Harbor — Claude Code (gen-only, disable_verif+capture) · high reasoning · Margin not reported · 420 reported samples | Google | 39.10% |
| 12 | [Opus 4.8](https://www.mercor.com/apex/oss-benchmarks/oss-redlinebench/) | RedlineBench Harbor — Claude Code (gen-only, disable_verif+capture) · max reasoning · Margin not reported · 420 reported samples | Anthropic | 38.20% |

## FAQ

### What does RedlineBench (Mercor) measure?

Negotiating and revising contracts across multiple turns. This table shows Mercor-run configurations for reference and is excluded from model rankings.

### Which model leads the published RedlineBench (Mercor) snapshot?

GPT-5.6 Sol currently leads the published RedlineBench (Mercor) snapshot with a score of 46.80%.

### How many models are evaluated on RedlineBench (Mercor)?

The October 15, 2026 capture contains 12 source configurations.

### Does RedlineBench (Mercor) affect BenchLM's overall score?

Not directly. RedlineBench (Mercor) is still displayed on BenchLM for reference, but it is excluded from the weighted scoring formula.