NL2Repo
We show this table for reference; we do not rank on it.
A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.
Benchmark score on NL2Repo — November 13, 2026
We compile the NL2Repo rows from provider self-reports and secondary reports. DeepSeek V4.1 Flash 1017 leads the table at 68.2%, followed by ERNIE 5.5 Flash Preview (64.8%) and DeepSeek V4 Pro 0813 (61.5%). We do not use these results to rank models overall.
DeepSeek V4.1 Flash 1017
DeepSeek
Reported by DeepSeekERNIE 5.5 Flash Preview
Baidu
Reported by BaiduDeepSeek V4 Pro 0813
DeepSeek
Reported by DeepSeek36 modelsCodingCurrentDisplay onlyUpdated November 13, 2026
Benchmark score table (36 models)
None of the 36 results on this page has a documented run by the benchmark or an independent evaluator: 33 reported by the labs that built the models and 3 reported in another publisher’s document.
| Rank | Model / configuration | Score | Who ran it | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | DeepSeek V4.1 Flash 1017DeepSeekReported by DeepSeek | 68.2% | Reported by DeepSeek | Not reported | Open |
| 2 | ERNIE 5.5 Flash PreviewBaiduReported by Baidu | 64.8% | Reported by Baidu | Not reported | Pending |
| 3 | DeepSeek V4 Pro 0813DeepSeekReported by DeepSeek | 61.5% | Reported by DeepSeek | Not reported | Open |
| 4 | MiniMax M3.1 FlashMiniMaxReported by MiniMax | 61.2% | Reported by MiniMax | Not reported | Closed |
| 5 | LongCat-2.5MeituanReported by Meituan | 60.5% | Reported by Meituan | Not reported | Open |
| 6 | Ornith-1.5-397BOrnith AIReported by Ornith AI | 59.5% | Reported by Ornith AI | Not reported | Open |
| 7 | Hy4 previewTencentReported by Tencent | 58.9% | Reported by Tencent | Not reported | Open |
| 8 | GLM-5.3Z.AIReported by Z.AI | 58% | Reported by Z.AI | Not reported | Open |
| 9 | GLM-5.3-FlashZ.AIReported by Z.AI | 56.3% | Reported by Z.AI | Not reported | Open |
| 10 | Qwen3.8 MaxAlibabaReported by Alibaba | 55.9% | Reported by Alibaba | Not reported | Open |
| 11 | DeepSeek V4 Flash 0731DeepSeekReported by DeepSeek | 54.2% | Reported by DeepSeek | Not reported | Open |
| 12 | dots3-note PreviewDots StudioReported by Dots Studio | 49.8% | Reported by Dots Studio | Not reported | Open |
| 13 | GLM-5.2Z.AIReported by Z.AI | 48.9% | Reported by Z.AI | Not reported | Open |
| 14 | Qwen3.8-Omni-FlashAlibabaReported by Alibaba | 48.9% | Reported by Alibaba | Not reported | Closed |
| 15 | Ornith-1.0-397BDeepReinforce AIReported by DeepReinforce AI | 48.2% | Reported by DeepReinforce AI | Not reported | Open |
| 16 | Qwen3.8-Flash-NextAlibabaReported by Alibaba | 48.1% | Reported by Alibaba | Not reported | Open |
| 17 | Qwen3.7 MaxAlibabaReported by Alibaba | 47.2% | Reported by Alibaba | Not reported | Closed |
| 18 | Seed 2.1 ProByteDanceReported by ByteDance | 47% | Reported by ByteDance | Not reported | Closed |
| 19 | Ornith-1.5-35B-A3BOrnith AIReported by Ornith AI | 46.2% | Reported by Ornith AI | Not reported | Open |
| 20 | Seed 2.1 TurboByteDanceReported by ByteDance | 43.7% | Reported by ByteDance | Not reported | Closed |
| 21 | Claude Opus 4.5AnthropicAnother lab’s comparison table | 43.2% | Another lab’s comparison table | Not reported | Closed |
| 22 | Qwen 3.6 Max (preview)AlibabaReported by Alibaba | 42.9% | Reported by Alibaba | Not reported | Closed |
| 23 | GLM-5.1Z.AIReported by Z.AI | 42.7% | Reported by Z.AI | Not reported | Open |
| 24 | Qwen3.8-27BAlibabaReported by Alibaba | 42.3% | Reported by Alibaba | Not reported | Open |
| 25 | MiniMax M3MiniMaxReported by MiniMax | 42.1% | Reported by MiniMax | Not reported | Open |
| 26 | Qwen3.7 PlusAlibabaReported by Alibaba | 41.1% | Reported by Alibaba | Not reported | Closed |
| 27 | MiniMax M2.7MiniMaxReported by MiniMax | 39.8% | Reported by MiniMax | Not reported | Open |
| 28 | Qwen3.6 PlusAlibabaAnother lab’s comparison table | 37.9% | Another lab’s comparison table | Not reported | Closed |
| 29 | Qwen3.6-27BAlibabaReported by Alibaba | 36.2% | Reported by Alibaba | Not reported | Open |
| 30 | GLM-5Z.AIReported by Z.AI | 35.9% | Reported by Z.AI | Not reported | Open |
| 31 | Ornith-1.0-35BDeepReinforce AIReported by DeepReinforce AI | 34.6% | Reported by DeepReinforce AI | Not reported | Open |
| 32 | Ornith-1.5-9BOrnith AIReported by Ornith AI | 32.4% | Reported by Ornith AI | Not reported | Open |
| 33 | Qwen3.5 397BAlibabaReported by Alibaba | 32.2% | Reported by Alibaba | Not reported | Open |
| 34 | Kimi K2.5Moonshot AIAnother lab’s comparison table | 32% | Another lab’s comparison table | Not reported | Open |
| 35 | Qwen3.6-35B-A3BAlibabaReported by Alibaba | 29.4% | Reported by Alibaba | Not reported | Open |
| 36 | Ornith-1.0-9BDeepReinforce AIReported by DeepReinforce AI | 27.2% | Reported by DeepReinforce AI | Not reported | Open |
- Lab-reported
- Benchmark-run
- Independent
Among the reported NL2Repo rows, DeepSeek V4.1 Flash 1017 is first at 68.2%. The third row is 6.7 points behind. The broader top-10 range is 12.3 points, so the table still separates the published systems.
36 models have been evaluated on NL2Repo. The benchmark falls in the Coding category. NL2Repo is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About NL2Repo
Year
2026
Tasks
Natural language to repository tasks
Format
Repository understanding benchmark
Difficulty
System-level software comprehension
MiniMax cites NL2Repo as a system-level engineering benchmark that rewards deep understanding of complex repositories and their operational structure.
Freshness and provenance
Version
NL2Repo 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does NL2Repo measure?
A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.
Which model scores highest on NL2Repo?
DeepSeek V4.1 Flash 1017 by DeepSeek currently leads with a score of 68.2% on NL2Repo.
How many models are evaluated on NL2Repo?
36 AI models have published results on NL2Repo in the BenchLM catalog.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.