Skip to main content
BenchLM

NL2Repo

We show this table for reference; we do not rank on it.

Data verified 34 confirmed releases in the last 30 daysFollow model changes

A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.

Benchmark score on NL2Repo — September 27, 2026

We compile the NL2Repo rows from provider self-reports and secondary reports. DeepSeek V4.1 Flash leads the table at 65.4%, followed by DeepSeek V4 Pro 0813 (61.5%) and Ornith-1.5-397B (59.5%). We do not use these results to rank models overall.

1Open weights

DeepSeek V4.1 Flash

DeepSeek

65.4%
Context 1M
2Open weights

DeepSeek V4 Pro 0813

DeepSeek

61.5%
Overall 63.48Context 1M
3Open weights

Ornith-1.5-397B

Ornith AI

59.5%
Overall 61.76Context 262K

29 modelsCodingCurrentDisplay onlyUpdated September 27, 2026

Benchmark score table (29 models)

Score
1
DeepSeek V4.1 FlashDeepSeek · Open weight
65.4%
2
DeepSeek V4 Pro 0813DeepSeek · Open weight
61.5%
3
Ornith-1.5-397BOrnith AI · Open weight
59.5%
4
Hy4 previewTencent · Open weight
58.9%
5
GLM-5.3Z.AI · Open weight
58%
6
GLM-5.3-FlashZ.AI · Open weight
56.3%
7
Qwen3.8 MaxAlibaba · Open weight
55.9%
8
DeepSeek V4 Flash 0731DeepSeek · Open weight
54.2%
9
dots3-note PreviewDots Studio · Open weight
49.8%
10
GLM-5.2Z.AI · Open weight
48.9%
11
Qwen3.8-Omni-FlashAlibaba · Closed
48.9%
12
Ornith-1.0-397BDeepReinforce AI · Open weight
48.2%
13
Qwen3.8-Flash-NextAlibaba · Open weight
48.1%
14
Qwen3.7 MaxAlibaba · Closed
47.2%
15
Seed 2.1 ProByteDance · Closed
47%
16
Ornith-1.5-35B-A3BOrnith AI · Open weight
46.2%
17
Seed 2.1 TurboByteDance · Closed
43.7%
18
Claude Opus 4.5Anthropic · Closed
43.2%
19
Qwen 3.6 Max (preview)Alibaba · Closed
42.9%
20
GLM-5.1Z.AI · Open weight
42.7%
21
Qwen3.8-27BAlibaba · Open weight
42.3%
22
MiniMax M3MiniMax · Open weight
42.1%
23
Qwen3.7 PlusAlibaba · Closed
41.1%
24
MiniMax M2.7MiniMax · Open weight
39.8%
25
Qwen3.6-27BAlibaba · Open weight
36.2%
26
Ornith-1.0-35BDeepReinforce AI · Open weight
34.6%
27
Ornith-1.5-9BOrnith AI · Open weight
32.4%
28
Qwen3.6-35B-A3BAlibaba · Open weight
29.4%
29
Ornith-1.0-9BDeepReinforce AI · Open weight
27.2%

Among the reported NL2Repo rows, DeepSeek V4.1 Flash is first at 65.4%. The third row is 5.9 points behind. The broader top-10 range is 16.5 points, so the table still separates the published systems.

29 models have been evaluated on NL2Repo. The benchmark falls in the Coding category. NL2Repo is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.

About NL2Repo

Year

2026

Tasks

Natural language to repository tasks

Format

Repository understanding benchmark

Difficulty

System-level software comprehension

MiniMax cites NL2Repo as a system-level engineering benchmark that rewards deep understanding of complex repositories and their operational structure.

MiniMax M2.7: Early Echoes of Self-Evolution

Freshness and provenance

Version

NL2Repo 2026

Refresh cadence

Quarterly

Staleness state

Current

Question availability

Public benchmark set

CurrentDisplay only

BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.

Questions

What does NL2Repo measure?

A repository-understanding benchmark that measures whether models can map natural-language requests onto the right code locations and system changes.

Which model scores highest on NL2Repo?

DeepSeek V4.1 Flash by DeepSeek currently leads with a score of 65.4% on NL2Repo.

How many models are evaluated on NL2Repo?

29 AI models have been evaluated on NL2Repo on BenchLM.

Compare top models on NL2Repo

DeepSeek V4.1 Flash vs DeepSeek V4 Pro 0813DeepSeek V4 Pro 0813 vs Ornith-1.5-397BOrnith-1.5-397B vs Hy4 previewHy4 preview vs GLM-5.3
Last updated: September 27, 2026 · BenchLM version NL2Repo 2026

Know when it’s worth switching models

The model to choose, the cheaper alternative, and the release we would wait on.

Read a sample issue

Join 2,000+ readers.

One email each week. Unsubscribe anytime.