xdev-bench-hard (Internal, pass@3) (xdev-bench-hard)
We show this table for reference; we do not rank on it.
Moonshot AI's internal benchmark of expert-curated, source-code-based vulnerability discovery and exploitation tasks, most reproducing recently disclosed vulnerabilities with mitigations enabled.
Benchmark score on xdev-bench-hard — October 29, 2026
We compile the xdev-bench-hard rows from provider self-reports. Kimi K3.1 Pro leads the table at 81.0%. We do not use these results to rank models overall.
1 modelAgenticCurrentDisplay onlyUpdated October 29, 2026
Benchmark score table (1 model)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Kimi K3.1 ProMoonshot AI | 81.0% | Not reported | Pending |
About xdev-bench-hard
Year
2026
Tasks
Vulnerability exploitation tasks
Format
Pass@3
Difficulty
Offensive security research
Each task provides vulnerable source code, a short bug hint, and the binary; the agent must achieve arbitrary code execution in a separate sandbox and retrieve a secret string. Moonshot reports pass@3 over a random subset of runs with no hard timeouts or output limits. BenchLM stores the provider-published internal benchmark value as display-only launch evidence and does not treat it as an independently reproducible score.
Freshness and provenance
Version
xdev-bench-hard 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does xdev-bench-hard measure?
Moonshot AI's internal benchmark of expert-curated, source-code-based vulnerability discovery and exploitation tasks, most reproducing recently disclosed vulnerabilities with mitigations enabled.
Which model scores highest on xdev-bench-hard?
Kimi K3.1 Pro by Moonshot AI currently leads with a score of 81.0% on xdev-bench-hard.
How many models are evaluated on xdev-bench-hard?
1 AI models have published results on xdev-bench-hard in the BenchLM catalog.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 6,100+ subscribers.
One email each week. Unsubscribe anytime.