NYT Connections LLM Benchmark (extended version) (NYT Connections)
We show this table for reference; we do not rank on it.
Lech Mazur's NYT Connections benchmark gives a model 940 puzzles from the New York Times word game, each with up to four extra trick words, and asks it to sort the words into four groups of four in one try. It tests word knowledge, wordplay and resisting plausible decoys.
Benchmark score on NYT Connections — November 13, 2026
We compile the NYT Connections rows from benchmark-owner or independent runs. Gemini 4 Carbon leads the table at 99.3%, followed by Gemini 4 Argon (98.6%) and GPT-6 Astra (98.4%). We do not use these results to rank models overall.
Gemini 4 Carbon
Gemini 4 Argon
GPT-6 Astra
OpenAI
Run by NYT Connections97 modelsReasoningRefreshingDisplay onlyUpdated November 13, 2026
Benchmark score table (97 models)
| Rank | Model / configuration | Score | Who ran it | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | Gemini 4 CarbonGoogleRun by NYT Connections | 99.3% | Run by NYT Connections | Not reported | Closed |
| 2 | Gemini 4 ArgonGoogleRun by NYT Connections | 98.6% | Run by NYT Connections | Not reported | Closed |
| 3 | GPT-6 AstraOpenAIRun by NYT Connections | 98.4% | Run by NYT Connections | Not reported | Closed |
| 4 | GPT-6.1 SolOpenAIRun by NYT Connections | 97.8% | Run by NYT Connections | Not reported | Closed |
| 5 | Gemini 3.1 ProGoogleRun by NYT Connections | 97.4% | Run by NYT Connections | Not reported | Closed |
| 6 | Gemini 3.8 FlashGoogleRun by NYT Connections | 97.4% | Run by NYT Connections | Not reported | Closed |
| 7 | Claude Fable 5.2AnthropicRun by NYT Connections | 97.3% | Run by NYT Connections | Not reported | Closed |
| 8 | GPT-5.5OpenAIRun by NYT Connections | 96.2% | Run by NYT Connections | Not reported | Closed |
| 9 | Gemini 3 ProGoogleRun by NYT Connections | 94.4% | Run by NYT Connections | Not reported | Closed |
| 10 | Claude Opus 5AnthropicRun by NYT Connections | 94.3% | Run by NYT Connections | Not reported | Closed |
| 11 | Gemini 3.7 FlashGoogleRun by NYT Connections | 94.0% | Run by NYT Connections | Not reported | Closed |
| 12 | GPT-5.6 SolOpenAIRun by NYT Connections | 93.8% | Run by NYT Connections | Not reported | Closed |
| 13 | Kimi K3Moonshot AIRun by NYT Connections | 93.6% | Run by NYT Connections | Not reported | Open |
| 14 | Kimi K3.1 ProMoonshot AIRun by NYT Connections | 93.4% | Run by NYT Connections | Not reported | Open |
| 15 | Muse WonderMetaRun by NYT Connections | 93.4% | Run by NYT Connections | Not reported | Closed |
| 16 | DeepSeek V4.1 Flash 1017DeepSeekRun by NYT Connections | 92.8% | Run by NYT Connections | Not reported | Open |
| 17 | Claude Fable 5AnthropicRun by NYT Connections | 92.7% | Run by NYT Connections | Not reported | Closed |
| 18 | Gemini 3.5 FlashGoogleRun by NYT Connections | 92.6% | Run by NYT Connections | Not reported | Closed |
| 19 | Claude Opus 4.6 (Adaptive)AnthropicRun by NYT Connections | 92.1% | Run by NYT Connections | Not reported | Closed |
| 20 | GPT-5.4OpenAIRun by NYT Connections | 91.3% | Run by NYT Connections | Not reported | Closed |
| 21 | Claude Opus 4.8AnthropicRun by NYT Connections | 91.1% | Run by NYT Connections | Not reported | Closed |
| 22 | GPT-6 SolOpenAIRun by NYT Connections | 90.1% | Run by NYT Connections | Not reported | Closed |
| 23 | Claude Fable 5.1AnthropicRun by NYT Connections | 90.0% | Run by NYT Connections | Not reported | Closed |
| 24 | Grok 4.20 Multi-agentxAIRun by NYT Connections | 89.6% | Run by NYT Connections | Not reported | Closed |
| 25 | Gemini 3.6 FlashGoogleRun by NYT Connections | 89.0% | Run by NYT Connections | Not reported | Closed |
| 26 | Claude Opus 5.5AnthropicRun by NYT Connections | 88.5% | Run by NYT Connections | Not reported | Closed |
| 27 | Qwen3.8 MaxAlibabaRun by NYT Connections | 88.3% | Run by NYT Connections | Not reported | Open |
| 28 | Grok 4.1 Fast (Reasoning)xAIRun by NYT Connections | 87.4% | Run by NYT Connections | Not reported | Closed |
| 29 | Kimi K2.6Moonshot AIRun by NYT Connections | 87.2% | Run by NYT Connections | Not reported | Open |
| 30 | GPT-6.1 TerraOpenAIRun by NYT Connections | 86.8% | Run by NYT Connections | Not reported | Closed |
| 31 | Grok 4.20xAIRun by NYT Connections | 85.4% | Run by NYT Connections | Not reported | Closed |
| 32 | Qwen3.7 MaxAlibabaRun by NYT Connections | 85.1% | Run by NYT Connections | Not reported | Closed |
| 33 | Muse Spark 1.3MetaRun by NYT Connections | 85.1% | Run by NYT Connections | Not reported | Closed |
| 34 | Muse Spark 1.1MetaRun by NYT Connections | 84.9% | Run by NYT Connections | Not reported | Closed |
| 35 | GPT-5.2OpenAIRun by NYT Connections | 83.6% | Run by NYT Connections | Not reported | Closed |
| 36 | Hy4TencentRun by NYT Connections | 83.5% | Run by NYT Connections | Not reported | Pending |
| 37 | Gemini 3 FlashGoogleRun by NYT Connections | 83.1% | Run by NYT Connections | Not reported | Closed |
| 38 | Claude Sonnet 5.5AnthropicRun by NYT Connections | 80.5% | Run by NYT Connections | Not reported | Closed |
| 39 | Grok 4.6xAIRun by NYT Connections | 80.0% | Run by NYT Connections | Not reported | Closed |
| 40 | Grok 4.5xAIRun by NYT Connections | 79.9% | Run by NYT Connections | Not reported | Closed |
| 41 | GPT-5.2 ProOpenAIRun by NYT Connections | 79.3% | Run by NYT Connections | Not reported | Closed |
| 42 | Muse Spark 1.2MetaRun by NYT Connections | 79.2% | Run by NYT Connections | Not reported | Closed |
| 43 | GPT-5.6 TerraOpenAIRun by NYT Connections | 78.4% | Run by NYT Connections | Not reported | Closed |
| 44 | Muse Spark OpenMetaRun by NYT Connections | 78.4% | Run by NYT Connections | Not reported | Open |
| 45 | GLM-5.1Z.AIRun by NYT Connections | 77.7% | Run by NYT Connections | Not reported | Open |
| 46 | Grok 4.7xAIRun by NYT Connections | 76.8% | Run by NYT Connections | Not reported | Closed |
| 47 | Step 5 PreviewStepFunRun by NYT Connections | 76.6% | Run by NYT Connections | Not reported | Open |
| 48 | Claude Sonnet 5AnthropicRun by NYT Connections | 75.1% | Run by NYT Connections | Not reported | Closed |
| 49 | Qwen3.7 PlusAlibabaRun by NYT Connections | 74.8% | Run by NYT Connections | Not reported | Closed |
| 50 | GLM-5 (Reasoning)Z.AIRun by NYT Connections | 74.8% | Run by NYT Connections | Not reported | Open |
| 51 | GLM-5.2Z.AIRun by NYT Connections | 74.3% | Run by NYT Connections | Not reported | Open |
| 52 | GLM-5.3Z.AIRun by NYT Connections | 74.2% | Run by NYT Connections | Not reported | Open |
| 53 | Qwen 3.6 Max (preview)AlibabaRun by NYT Connections | 74.1% | Run by NYT Connections | Not reported | Closed |
| 54 | Gemma 4 31BGoogleRun by NYT Connections | 70.6% | Run by NYT Connections | Not reported | Open |
| 55 | Kimi K2.5 (Reasoning)Moonshot AIRun by NYT Connections | 69.9% | Run by NYT Connections | Not reported | Open |
| 56 | GPT-5.6 LunaOpenAIRun by NYT Connections | 69.4% | Run by NYT Connections | Not reported | Closed |
| 57 | GPT-6 LunaOpenAIRun by NYT Connections | 68.7% | Run by NYT Connections | Not reported | Closed |
| 58 | Hy4 previewTencentRun by NYT Connections | 68.2% | Run by NYT Connections | Not reported | Open |
| 59 | Claude Haiku 5.5AnthropicRun by NYT Connections | 65.7% | Run by NYT Connections | Not reported | Closed |
| 60 | MiniMax M3MiniMaxRun by NYT Connections | 65.1% | Run by NYT Connections | Not reported | Open |
| 61 | GPT-5.4 miniOpenAIRun by NYT Connections | 61.8% | Run by NYT Connections | Not reported | Closed |
| 62 | Gemini 3.5 Flash-LiteGoogleRun by NYT Connections | 60.4% | Run by NYT Connections | Not reported | Closed |
| 63 | Qwen3.6 PlusAlibabaRun by NYT Connections | 60.3% | Run by NYT Connections | Not reported | Closed |
| 64 | Qwen3.5 397B (Reasoning)AlibabaRun by NYT Connections | 58.9% | Run by NYT Connections | Not reported | Open |
| 65 | Grok 4.3xAIRun by NYT Connections | 55.2% | Run by NYT Connections | Not reported | Closed |
| 66 | Qwen3.8-27BAlibabaRun by NYT Connections | 54.5% | Run by NYT Connections | Not reported | Open |
| 67 | Claude Opus 4.5 ThinkingAnthropicRun by NYT Connections | 52.5% | Run by NYT Connections | Not reported | Closed |
| 68 | Qwen3.5-122B-A10BAlibabaRun by NYT Connections | 51.7% | Run by NYT Connections | Not reported | Open |
| 69 | Claude Opus 4.5AnthropicRun by NYT Connections | 49.4% | Run by NYT Connections | Not reported | Closed |
| 70 | Qwen3.5-27BAlibabaRun by NYT Connections | 47.9% | Run by NYT Connections | Not reported | Open |
| 71 | Claude Sonnet 4.6AnthropicRun by NYT Connections | 44.8% | Run by NYT Connections | Not reported | Closed |
| 72 | Qwen3.7 FlashAlibabaRun by NYT Connections | 43.8% | Run by NYT Connections | Not reported | Closed |
| 73 | Claude Opus 4.6AnthropicRun by NYT Connections | 43.2% | Run by NYT Connections | Not reported | Closed |
| 74 | Qwen3.6-35B-A3BAlibabaRun by NYT Connections | 41.6% | Run by NYT Connections | Not reported | Open |
| 75 | Hy3TencentRun by NYT Connections | 41.2% | Run by NYT Connections | Not reported | Open |
| 76 | Step 3.7 FlashStepFunRun by NYT Connections | 39.7% | Run by NYT Connections | Not reported | Open |
| 77 | Claude Opus 4.7 (Adaptive)AnthropicRun by NYT Connections | 39.0% | Run by NYT Connections | Not reported | Closed |
| 78 | Claude Sonnet 4.5 ThinkingAnthropicRun by NYT Connections | 37.3% | Run by NYT Connections | Not reported | Closed |
| 79 | DeepSeek V3.2 (Thinking)DeepSeekRun by NYT Connections | 36.7% | Run by NYT Connections | Not reported | Open |
| 80 | Claude Sonnet 4.5AnthropicRun by NYT Connections | 35.8% | Run by NYT Connections | Not reported | Closed |
| 81 | MiMo-V2.5-ProXiaomiRun by NYT Connections | 34.4% | Run by NYT Connections | Not reported | Closed |
| 82 | Step 3.5 FlashStepFunRun by NYT Connections | 28.4% | Run by NYT Connections | Not reported | Open |
| 83 | MiniMax M3.1 FlashMiniMaxRun by NYT Connections | 28.3% | Run by NYT Connections | Not reported | Closed |
| 84 | Mistral Large 4MistralRun by NYT Connections | 27.4% | Run by NYT Connections | Not reported | Open |
| 85 | MiMo-V2-ProXiaomiRun by NYT Connections | 25.8% | Run by NYT Connections | Not reported | Closed |
| 86 | MiniMax M2.7MiniMaxRun by NYT Connections | 24.7% | Run by NYT Connections | Not reported | Open |
| 87 | Hy3 PreviewTencentRun by NYT Connections | 17.2% | Run by NYT Connections | Not reported | Open |
| 88 | MiniMax M2.5MiniMaxRun by NYT Connections | 16.8% | Run by NYT Connections | Not reported | Closed |
| 89 | Trinity-Large-ThinkingArcee AIRun by NYT Connections | 16.5% | Run by NYT Connections | Not reported | Open |
| 90 | Claude Haiku 4.5AnthropicRun by NYT Connections | 14.3% | Run by NYT Connections | Not reported | Closed |
| 91 | Mistral Medium 3.5 128BMistralRun by NYT Connections | 12.9% | Run by NYT Connections | Not reported | Open |
| 92 | Grok 4.1 FastxAIRun by NYT Connections | 12.3% | Run by NYT Connections | Not reported | Closed |
| 93 | Claude Opus 4.7AnthropicRun by NYT Connections | 10.8% | Run by NYT Connections | Not reported | Closed |
| 94 | Gemini 3.1 Flash-LiteGoogleRun by NYT Connections | 8.2% | Run by NYT Connections | Not reported | Closed |
| 95 | DeepSeek V3.2DeepSeekRun by NYT Connections | 8.2% | Run by NYT Connections | Not reported | Open |
| 96 | Llama 4 MaverickMetaRun by NYT Connections | 8.0% | Run by NYT Connections | Not reported | Open |
| 97 | Mistral Large 3MistralRun by NYT Connections | 7.5% | Run by NYT Connections | Not reported | Open |
- Lab-reported
- Benchmark-run
- Independent
How to read this leaderboard
Higher is better. Gemini 4 Argon leads at 98.6%, with GPT-6 Astra at 98.4% and GPT-6.1 Sol at 97.8%, so the top of the board is close to solved. Reasoning effort matters a lot: GPT-5.2 scores 83.6% at xhigh and 14.5% with reasoning off. The best open-weight runs are Kimi K3 at 93.6% and Kimi K3.1 Pro at 93.4%. Because a full solve counts far more than a partial one, a few points near the top mean several more puzzles solved outright.
Operator receipt: 97 sourced rows are currently displayable on this page; the leading published row is Gemini 4 Carbon at 99.3%.
Honest limit: The puzzles come from a published game, so older puzzles and their answers may be in training data; the newest-100 board helps with that but does not rule it out. The scoring rule changed in July 2026 and the extended version's puzzle count grew over time (601, 651, 759 and, since February 2026, 940), so older published scores are not comparable with today's. Refused or blocked puzzles count as zero. One author runs every model, and the prompts are fixed rather than tuned per model. It measures verbal association and constraint solving, not general intelligence. The benchmark is not affiliated with the New York Times. We show it as display-only; it does not enter overall or category rankings.
Among the reported NYT Connections rows, Gemini 4 Carbon is first at 99.3%. The third row is 0.9 points behind. The broader top-10 range is 5.0 points, so many of the published results sit in a relatively narrow band.
97 models have been evaluated on NYT Connections. The benchmark falls in the Reasoning category. We keep these results separate from the weighted global scoring system, so they remain source-specific evidence. NYT Connections is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About NYT Connections
Year
2024
Tasks
940 NYT Connections puzzles, each with up to four added trick words
Format
Sort 16 to 20 words into four groups of four, one attempt per puzzle, no feedback
Difficulty
Wordplay, word knowledge and decoys that fit almost-but-not-quite
Each puzzle is a Connections grid with up to four added words that fit none of its categories; the author checks that no added word fits any group in that puzzle. The model gets one attempt per puzzle, with no feedback and no "one away" hint, where a person on the Times site gets four mistakes. Since July 2026 the headline score uses quadratic-v1 scoring: a puzzle with g exact groups scores (g / 4) squared, so 0 to 4 groups earn 0%, 6.25%, 25%, 56.25% and 100%, and the score is the mean over all puzzles. Only runs that completed all 940 puzzles are on the main board. Many models are listed at several reasoning settings; we store one run per model on its profile (normally the highest setting) and name the others in its row note. The author also publishes a board for the newest 100 puzzles, as a check on puzzles that may sit in training data.
Freshness and provenance
Version
NYT Connections 2024
Refresh cadence
Annual
Staleness state
Refreshing
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does the NYT Connections benchmark measure?
Whether a model can solve the New York Times Connections word puzzle: sort the words into four groups of four that share a hidden link. This version adds up to four trick words per puzzle that fit no group, and allows one attempt with no feedback, so it rewards wordplay, broad word knowledge and resisting near-misses.
Which model leads NYT Connections?
Gemini 4 Argon at high reasoning leads the extended version with 98.6%, ahead of GPT-6 Astra at max reasoning (98.4%) and GPT-6.1 Sol at max reasoning (97.8%). The best open-weight model is Kimi K3 at 93.6%.
Why do some models appear several times?
The author runs many models at several reasoning settings, and the setting changes the score a lot: GPT-5.2 scores 83.6% at xhigh reasoning and 14.5% with reasoning off. Each model profile on BenchLM shows one run, usually the highest setting, and its note lists the other runs.
Compare top models on NYT Connections
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.