CVDP — Mercor evaluation (CVDP (Mercor))
We show this table for reference; we do not rank on it.
Hardware design, verification, and comprehension tasks. This table shows Mercor-run configurations for reference and is excluded from model rankings.
Evaluation results published by Mercor. Published evaluation aggregates only; task contents and dataset license grants are not included.
Pass@1 on CVDP (Mercor) — October 15, 2026 capture
We mirror the published pass@1 view for CVDP (Mercor). Opus 5.5 leads the public snapshot at 64.10%, followed by Fable 5.1 (62.60%) and Opus 5 (62.20%). We do not use these results to rank models overall.
Opus 5.5
Anthropic
Terminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoning
Published margin ±4 · 2,575 reported samples
Fable 5.1
Anthropic
Terminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoning
Published margin ±3.9 · 2,585 reported samples
Opus 5
Anthropic
Terminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoning
Published margin ±3.9 · 2,585 reported samples
25 configurationsCodingDated Mercor configurationsDisplay onlyUpdated October 15, 2026 capture
Pass@1 table (25 configurations)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Opus 5.5AnthropicTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±4 · 2,575 reported samples | 64.10% | Not reported | Unknown |
| 2 | Fable 5.1AnthropicTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±3.9 · 2,585 reported samples | 62.60% | Not reported | Unknown |
| 3 | Opus 5AnthropicTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±3.9 · 2,585 reported samples | 62.20% | Not reported | Unknown |
| 4 | Fable 5.1AnthropicTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · high reasoningPublished margin ±3.9 · 2,585 reported samples | 61.20% | Not reported | Unknown |
| 5 | Sonnet 5.5AnthropicTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±4 · 2,568 reported samples | 60.60% | Not reported | Unknown |
| 6 | Grok 4.6xAITerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · xhigh reasoningPublished margin ±3.9 · 2,585 reported samples | 58.00% | Not reported | Unknown |
| 7 | Haiku 5.5AnthropicTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±4 · 2,576 reported samples | 57.00% | Not reported | Unknown |
| 8 | Sonnet 5AnthropicTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±4 · 2,541 reported samples | 56.40% | Not reported | Unknown |
| 9 | Kimi K3KimiTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±3.9 · 2,585 reported samples | 55.50% | Not reported | Unknown |
| 10 | GPT-5.6 SolOpenAITerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±4 · 2,585 reported samples | 55.40% | Not reported | Unknown |
| 11 | Qwen3.8-MaxAlibabaTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · xhigh reasoningPublished margin ±4.1 · 2,518 reported samples | 55.20% | Not reported | Unknown |
| 12 | GPT-6.1 SolOpenAITerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±4.2 · 2,567 reported samples | 55.10% | Not reported | Unknown |
| 13 | GPT-6 AstraOpenAITerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±4.2 · 2,585 reported samples | 54.90% | Not reported | Unknown |
| 14 | Grok 4.7xAITerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · xhigh reasoningPublished margin ±3.9 · 2,549 reported samples | 54.90% | Not reported | Unknown |
| 15 | Muse Spark 1.3MetaTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · xhigh reasoningPublished margin ±3.9 · 2,585 reported samples | 54.30% | Not reported | Unknown |
| 16 | GPT-5.6 SolOpenAITerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · xhigh reasoningPublished margin ±4.1 · 2,585 reported samples | 54.10% | Not reported | Unknown |
| 17 | DeepSeek-V4-Pro-0813DeepSeekTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±3.9 · 2,585 reported samples | 53.10% | Not reported | Unknown |
| 18 | GLM-5.3ZhipuTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±3.9 · 2,585 reported samples | 53.10% | Not reported | Unknown |
| 19 | GPT-6 SolOpenAITerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±4 · 2,585 reported samples | 52.70% | Not reported | Unknown |
| 20 | Gemini 3.1 ProGoogleTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · high reasoningPublished margin ±4 · 2,585 reported samples | 51.30% | Not reported | Unknown |
| 21 | GPT-5.6 TerraOpenAITerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±3.8 · 2,585 reported samples | 50.90% | Not reported | Unknown |
| 22 | GPT-6 LunaOpenAITerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±4 · 2,585 reported samples | 50.80% | Not reported | Unknown |
| 23 | Gemini 3.8 FlashGoogleTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · high reasoningPublished margin ±3.8 · 2,585 reported samples | 50.70% | Not reported | Unknown |
| 24 | GPT-5.6 LunaOpenAITerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · max reasoningPublished margin ±3.9 · 2,585 reported samples | 48.40% | Not reported | Unknown |
| 25 | MiniMax-M3MiniMaxTerminus via Harbor (1 hour time limit) for code generation; single-shot for comprehension · high reasoningPublished margin ±3.7 · 2,585 reported samples | 44.80% | Not reported | Unknown |
How to read this Mercor evaluation
We captured Mercor's CVDP results on October 15, 2026. The table preserves 25 published configurations, their source identifiers, reasoning settings, and disclosed pass@1 values. These are results from Mercor's evaluation setup; we did not rerun them.
This dated source table is display only and does not enter overall or category rankings. Source configurations retain their external identity, including Pro descriptors and different effort settings.
The headline is the source page's selected Pass@1 metric. The task set, harness, tools, and grader belong to this Mercor run and may differ from another evaluator's results.
The row labels retain published error margins and sample counts when supplied. Missing margins and counts remain unreported; a sample count is not used to reconstruct a task denominator. Except where the dedicated source defines an interval, the confidence level is unreported. Equal displayed scores and overlapping margins do not establish statistical ties. Capture time does not establish individual evaluation dates.
The general methodology lists code generation: programmatic — a cocotb testbench runs against icarus verilog in docker and the task passes iff pytest passes. comprehension: bleu-2 overlap against the reference answer (correspondence) and an llm rating (q&a). and a judge of None for code generation; BLEU/ROUGE similarity and an LLM judge for comprehension. Its run-count label is k = 5. The linked result page takes precedence where its setup differs, as noted above.
Snapshot
The published CVDP (Mercor) snapshot places Opus 5.5 first at 64.10%. The third row is 1.90 points behind. The broader top-10 range is 8.70 points, so many of the published results sit in a relatively narrow band.
25 configurations are shown for CVDP (Mercor). The benchmark falls in the Coding category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. CVDP (Mercor) is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About CVDP (Mercor)
Tasks
517 source-reported tasks
Format
Pass@1
Difficulty
Source-specific evaluation
The selected headline is Pass@1. Source model identifiers, effort settings, published margins, and reported sample counts remain attached to the source configuration. Public-set results and Mercor task extensions occupy separate tables.
Freshness and provenance
Version
CVDP; Mercor Pass@1
Refresh cadence
Reviewed dated source capture
Staleness state
Dated Mercor configurations
Question availability
Upstream task access and terms; only aggregate results mirrored
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does CVDP (Mercor) measure?
Hardware design, verification, and comprehension tasks. This table shows Mercor-run configurations for reference and is excluded from model rankings.
Which model leads the published CVDP (Mercor) snapshot?
Opus 5.5 currently leads the published CVDP (Mercor) snapshot with 64.10% pass@1. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on CVDP (Mercor)?
The October 15, 2026 capture snapshot contains 25 source configurations.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 5,600+ readers.
One email each week. Unsubscribe anytime.