WeirdML v3
We show this table for reference; we do not rank on it.
Håvard Tveit Ihle's WeirdML v3 gives coding agents 11 hand-made machine-learning and data-analysis tasks with unfamiliar data, unspecified goals or very limited feedback, and scores how well and how efficiently they improve as they spend tokens.
Official score on WeirdML v3 — WeirdML v3
We mirror the published official score view for WeirdML v3. Claude Fable 5.2 (xhigh) leads the public snapshot at 43.0%, followed by GPT-6 Astra (xhigh) (42.2%) and GPT-6.1 Sol (xhigh) (38.3%). We do not use these results to rank models overall.
1Closed
Claude Fable 5.2 (xhigh)
Anthropic
claude_code · xhigh reasoning
43.0%
2Closed
GPT-6 Astra (xhigh)
OpenAI
codex_cli · xhigh reasoning
42.2%
3Closed
GPT-6.1 Sol (xhigh)
OpenAI
codex_cli · xhigh reasoning
38.3%
29 modelsCodingCurrentDisplay onlyUpdated WeirdML v3
Official score table (29 models)
29 of 29 rows
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Claude Fable 5.2 (xhigh)Anthropicclaude_code · xhigh reasoning | 43.0% | Not reported | Closed |
| 2 | GPT-6 Astra (xhigh)OpenAIcodex_cli · xhigh reasoning | 42.2% | Not reported | Closed |
| 3 | GPT-6.1 Sol (xhigh)OpenAIcodex_cli · xhigh reasoning | 38.3% | Not reported | Closed |
| 4 | Claude Opus 5.5 (xhigh)Anthropicclaude_code · xhigh reasoning | 31.2% | Not reported | Closed |
| 5 | GPT-6.1 Terra (xhigh)OpenAIcodex_cli · xhigh reasoning | 28.1% | Not reported | Closed |
| 6 | Claude Fable 5.1 (xhigh)Anthropicclaude_code · xhigh reasoning | 26.0% | Not reported | Closed |
| 7 | Gemini 4 Argon (high)Googlegemini_cli · high reasoning | 20.3% | Not reported | Closed |
| 8 | Claude Sonnet 5.5 (xhigh)Anthropicclaude_code · xhigh reasoning | 19.9% | Not reported | Closed |
| 9 | GPT-6 Sol (xhigh)OpenAIcodex_cli · xhigh reasoning | 19.7% | Not reported | Closed |
| 10 | Claude Opus 5 (xhigh)Anthropicclaude_code · xhigh reasoning | 19.0% | Not reported | Closed |
| 11 | Claude Fable 5 (xhigh)Anthropicclaude_code · xhigh reasoning | 15.6% | Not reported | Closed |
| 12 | Muse Wonder (xhigh)Metacodex_cli · xhigh reasoning | 15.1% | Not reported | Closed |
| 13 | GPT-5.6 Sol (xhigh)OpenAIcodex_cli · xhigh reasoning | 15.0% | Not reported | Closed |
| 14 | Claude Haiku 5.5 (xhigh)Anthropicclaude_code · xhigh reasoning | 11.7% | Not reported | Closed |
| 15 | Kimi K3.1 Pro (xhigh, Fireworks)Moonshot AIkimi_code · xhigh reasoning | 9.9% | Not reported | Open |
| 16 | DeepSeek V4.1 Flash 1017 (high, InferenceNet)DeepSeekcodex_cli · high reasoning | 9.1% | Not reported | Open |
| 17 | Grok 4.7 (xhigh)xAIcodex_cli · xhigh reasoning | 8.5% | Not reported | Closed |
| 18 | GPT-6 Luna (xhigh)OpenAIcodex_cli · xhigh reasoning | 7.9% | Not reported | Closed |
| 19 | Kimi K3 (high, Fireworks)Moonshot AIkimi_code · high reasoning | 7.5% | Not reported | Open |
| 20 | Gemini 3.8 Flash (high)Googlegemini_cli · high reasoning | 7.4% | Not reported | Closed |
| 21 | Muse Spark 1.3 (xhigh)Metacodex_cli · xhigh reasoning | 6.7% | Not reported | Closed |
| 22 | Claude Opus 4.5 (high)Anthropicclaude_code · high reasoning | 6.5% | Not reported | Closed |
| 23 | DeepSeek V4.1 Flash (high, Novita)DeepSeekopencode · high reasoning | 5.9% | Not reported | Open |
| 24 | DeepSeek V4.1 Flash (high, Novita)DeepSeekcodex_cli · high reasoning | 5.8% | Not reported | Open |
| 25 | GPT-5.6 Luna (xhigh)OpenAIcodex_cli · xhigh reasoning | 5.2% | Not reported | Closed |
| 26 | GPT-5.2 (xhigh)OpenAIcodex_cli · xhigh reasoning | 5.0% | Not reported | Closed |
| 27 | Mistral Large 4 (high, Mistral)Mistralopencode · high reasoning | 4.2% | Not reported | Closed |
| 28 | Inkling (xhigh, Together)Unknownopencode · xhigh reasoning | 3.9% | Not reported | Open |
| 29 | Inkling (xhigh, DeepInfra)Unknownopencode · xhigh reasoning | 3.7% | Not reported | Open |
WeirdML
BenchLM mirrors WeirdML average-accuracy scores from the official CSV. WeirdML is excluded from BenchLM overall and category rankings.
162 of 162 rows
| Source rank | Source model or configuration | Average accuracy | Source setup | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | gpt-6-astra-pro (max)OpenAI | 93.57% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 2 | GPT-6 AstraOpenAI | 93.26% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 3 | Claude Fable 5.1Anthropic | 92.90% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 4 | GPT-6 AstraOpenAI | 92.87% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 5 | Claude Fable 5.1Anthropic | 92.30% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 6 | Claude Fable 5Anthropic | 91.94% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 7 | Claude Opus 5Anthropic | 91.78% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 8 | Claude Opus 5Anthropic | 91.59% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 9 | gpt-5.6-sol-pro (max)OpenAI | 89.43% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 10 | GPT-5.6 SolOpenAI | 88.76% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 11 | Claude Fable 5Anthropic | 87.85% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 12 | GPT-5.6 SolOpenAI | 86.97% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 13 | Claude Opus 5Anthropic | 86.34% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 14 | GPT-5.5OpenAI | 84.91% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 15 | Gemini 3.8 FlashGoogle | 84.81% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 16 | GPT-5.5OpenAI | 83.90% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 17 | Claude Opus 4.8Anthropic | 82.89% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 18 | Kimi K3Moonshot AI | 82.57% | Havard Tveit IhleOriginal result | Not reported | Open |
| 19 | GPT-5.6 TerraOpenAI | 78.27% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 20 | Claude Opus 4.6Anthropic | 77.95% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 21 | GPT-5.3 CodexOpenAI | 77.90% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 22 | GPT-5.4OpenAI | 77.70% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 23 | qwen3.8-2.4t-a95b (xhigh)Alibaba | 76.49% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 24 | Claude Opus 4.7Anthropic | 76.44% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 25 | Claude Opus 4.7Anthropic | 76.40% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 26 | Claude Opus 4.8Anthropic | 76.04% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 27 | Claude Opus 4.7Anthropic | 75.49% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 28 | GLM-5.3Z.AI | 75.40% | Havard Tveit IhleOriginal result | Not reported | Open |
| 29 | GPT-5.2OpenAI | 72.19% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 30 | gemini-3.1-pro-preview (high)Google | 72.07% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 31 | Claude Opus 4.8Anthropic | 70.45% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 32 | GLM-5.2Z.AI | 70.12% | Havard Tveit IhleOriginal result | Not reported | Open |
| 33 | gemini-3-pro-preview (high)Google | 69.93% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 34 | Claude Sonnet 5Anthropic | 68.98% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 35 | GLM-5.2Z.AI | 67.31% | Havard Tveit IhleOriginal result | Not reported | Open |
| 36 | Grok 4.6xAI | 67.29% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 37 | GPT-5.5OpenAI | 67.15% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 38 | DeepSeek V4 Pro 0813DeepSeek | 66.22% | Havard Tveit IhleOriginal result | Not reported | Open |
| 39 | Claude Sonnet 4.6Anthropic | 66.07% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 40 | Claude Opus 4.6Anthropic | 65.87% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 41 | Claude Opus 4.5Anthropic | 63.74% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 42 | GPT-5.2OpenAI | 63.44% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 43 | DeepSeek V4 Flash 0731DeepSeek | 62.96% | Havard Tveit IhleOriginal result | Not reported | Open |
| 44 | Gemini 3.5 FlashGoogle | 62.64% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 45 | gemini-3-flash-preview (high)Google | 61.60% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 46 | GPT-5.6 LunaOpenAI | 60.86% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 47 | GPT-5.1OpenAI | 60.77% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 48 | gpt-5 (high)OpenAI | 60.70% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 49 | gpt-5-pro (high)OpenAI | 60.39% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 50 | Muse Spark 1.2Meta | 60.38% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 51 | GPT-5.4 miniOpenAI | 60.30% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 52 | o3-proOpenAI | 58.21% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 53 | GPT-5.4OpenAI | 57.44% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 54 | GLM-5.1Z.AI | 57.10% | Havard Tveit IhleOriginal result | Not reported | Open |
| 55 | DeepSeek V4 Flash 0731DeepSeek | 57.05% | Havard Tveit IhleOriginal result | Not reported | Open |
| 56 | Gemini 3.6 FlashGoogle | 56.10% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 57 | kimi-k2.6Moonshot AI | 55.86% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 58 | gpt-5-codex (high)OpenAI | 54.53% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 59 | Kimi K2.7 CodeMoonshot AI | 54.12% | Havard Tveit IhleOriginal result | Not reported | Open |
| 60 | Gemini 2.5 ProGoogle | 54.03% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 61 | GPT-5 miniOpenAI | 52.67% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 62 | o4-mini (high)OpenAI | 52.56% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 63 | o3OpenAI | 52.42% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 64 | Grok 4.20xAI | 52.26% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 65 | gemma4-31bUnknown | 52.26% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 66 | gemini-3.1-flash-lite-preview (high)Google | 52.19% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 67 | Grok 4.3xAI | 49.89% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 68 | GPT-5.2OpenAI | 49.60% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 69 | GPT-5.4 nanoOpenAI | 49.23% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 70 | DeepSeek V4 Pro 0813DeepSeek | 48.61% | Havard Tveit IhleOriginal result | Not reported | Open |
| 71 | GLM-5Z.AI | 48.17% | Havard Tveit IhleOriginal result | Not reported | Open |
| 72 | GPT-OSS 120BOpenAI | 48.17% | Havard Tveit IhleOriginal result | Not reported | Open |
| 73 | claude-sonnet-4.5 (thinking 16k)Anthropic | 47.71% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 74 | o1-previewOpenAI | 47.56% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 75 | deepseek-v3.2-specialeDeepSeek | 46.73% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 76 | claude-sonnet-4.5 (no thinking)Anthropic | 46.70% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 77 | DeepSeek V4 Pro 0813DeepSeek | 46.53% | Havard Tveit IhleOriginal result | Not reported | Open |
| 78 | Grok 4.5xAI | 46.43% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 79 | Claude 4 SonnetAnthropic | 46.11% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 80 | o1OpenAI | 46.08% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 81 | claude-opus-4.1 (thinking 16k)Anthropic | 45.86% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 82 | grok-4-07-09xAI | 45.73% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 83 | DeepSeek V4 Flash 0731DeepSeek | 45.63% | Havard Tveit IhleOriginal result | Not reported | Open |
| 84 | Kimi K2.5Moonshot AI | 45.60% | Havard Tveit IhleOriginal result | Not reported | Open |
| 85 | claude-haiku-4.5 (no thinking)Anthropic | 45.40% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 86 | claude-haiku-4.5 (thinking 16k)Anthropic | 44.15% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 87 | Claude 4 SonnetAnthropic | 43.88% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 88 | DeepSeek V4 Flash 0731DeepSeek | 43.76% | Havard Tveit IhleOriginal result | Not reported | Open |
| 89 | o3-mini (high)OpenAI | 43.75% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 90 | Claude 4 OpusAnthropic | 43.72% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 91 | mistral-medium-3.5-128bMistral | 43.71% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 92 | nemotron-3-ultra-550b-a55bNVIDIA | 43.45% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 93 | Mercury 2Inception | 43.20% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 94 | grok-4-fastxAI | 42.86% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 95 | kimi-k2-thinkingMoonshot AI | 42.79% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 96 | Grok 3 MinixAI | 42.58% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 97 | gemini-2.5-flash-09-25 (thinking 16k)Google | 41.91% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 98 | GPT-OSS 120BOpenAI | 41.88% | Havard Tveit IhleOriginal result | Not reported | Open |
| 99 | DeepSeek-R1-0528DeepSeek | 41.63% | Havard Tveit IhleOriginal result | Not reported | Open |
| 100 | qwen3-coderAlibaba | 41.17% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 101 | qwen3-235b-a22b-thinking-2507Alibaba | 41.04% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 102 | Gemini 2.5 FlashGoogle | 40.95% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 103 | GPT-OSS 20BOpenAI | 40.93% | Havard Tveit IhleOriginal result | Not reported | Open |
| 104 | GLM-4.5Z.AI | 40.58% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 105 | Claude 3.5 SonnetAnthropic | 39.97% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 106 | gpt-5-chatOpenAI | 39.77% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 107 | qwen3.5-27bAlibaba | 39.53% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 108 | deepseek-v3.2-exp (thinking)DeepSeek | 39.48% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 109 | gpt-4.5OpenAI | 39.37% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 110 | Kimi K2Moonshot AI | 39.36% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 111 | GPT-4.1OpenAI | 39.04% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 112 | Gemini 3.5 Flash-LiteGoogle | 39.00% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 113 | qwen3-235b-a22b-07-25Alibaba | 38.70% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 114 | deepseek-chat-v3.1 (thinking)DeepSeek | 38.36% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 115 | GPT-5 nanoOpenAI | 38.06% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 116 | nemotron-3-super-120b-a12bNVIDIA | 38.01% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 117 | GPT-5.4 nanoOpenAI | 37.96% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 118 | GPT-5.4 miniOpenAI | 37.90% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 119 | GPT-4.1 miniOpenAI | 37.61% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 120 | deepseek-chat-v3.1DeepSeek | 37.50% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 121 | Qwen3 235B-A22BAlibaba | 37.28% | Havard Tveit IhleOriginal result | Not reported | Open |
| 122 | grok-3xAI | 37.24% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 123 | MiniMax M2.7MiniMax | 36.95% | Havard Tveit IhleOriginal result | Not reported | Open |
| 124 | GPT-OSS 20BOpenAI | 36.83% | Havard Tveit IhleOriginal result | Not reported | Open |
| 125 | kimi-k2-0905Moonshot AI | 36.68% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 126 | DeepSeek-R1DeepSeek | 36.49% | Havard Tveit IhleOriginal result | Not reported | Open |
| 127 | o1-miniOpenAI | 36.32% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 128 | deepseek-v3-0324DeepSeek | 36.08% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 129 | gemini-2.5-flash-lite-preview-06-17 (thinking 16k)Google | 35.22% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 130 | gemma4-26b-a4bUnknown | 35.17% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 131 | Grok Code Fast 1xAI | 35.06% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 132 | qwen3.6-35b-a3bAlibaba | 34.49% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 133 | qwen3-coder-nextAlibaba | 34.40% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 134 | mistral-medium-3.1Mistral | 33.13% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 135 | InklingThinking Machines Lab | 32.29% | Havard Tveit IhleOriginal result | Not reported | Open |
| 136 | Claude 3.5 SonnetAnthropic | 30.97% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 137 | Claude 3.5 HaikuAnthropic | 30.73% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 138 | qwen3-30b-a3b (thinking)Alibaba | 29.75% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 139 | GPT-5 nanoOpenAI | 25.89% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 140 | gemini-2.0-flash-001Google | 25.77% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 141 | GPT-4oOpenAI | 25.12% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 142 | gemini-flash-1.5-002Google | 24.87% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 143 | Llama 4 MaverickMeta | 24.47% | Havard Tveit IhleOriginal result | Not reported | Open |
| 144 | grok-2-1212xAI | 22.24% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 145 | gemini-pro-1.5-002Google | 22.20% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 146 | llama-3.1-405b-instructMeta | 21.38% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 147 | Claude 3 OpusAnthropic | 19.22% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 148 | GPT-4.1 nanoOpenAI | 18.98% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 149 | GPT-4 TurboOpenAI | 18.01% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 150 | qwen-2.5-72b-instructAlibaba | 15.97% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 151 | llama-3.3-70b-instructMeta | 14.44% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 152 | deepseek-r1-0528-qwen3-8bAlibaba | 12.99% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 153 | gpt-4-0613OpenAI | 12.36% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 154 | GPT-4o miniOpenAI | 11.76% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 155 | qwen2-72b-instructAlibaba | 11.30% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 156 | claude-3-sonnetAnthropic | 10.16% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 157 | Claude 3 HaikuAnthropic | 9.84% | Havard Tveit IhleOriginal result | Not reported | Closed |
| 158 | llama-3.1-70b-instructMeta | 8.97% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 159 | claude-2.1Anthropic | 7.06% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 160 | gpt-3.5-turbo-0125OpenAI | 3.48% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 161 | mixtral-8x22b-instructUnknown | 3.17% | Havard Tveit IhleOriginal result | Not reported | Unknown |
| 162 | llama-3.1-8b-instructMeta | 1.73% | Havard Tveit IhleOriginal result | Not reported | Unknown |
WeirdML v1
BenchLM mirrors the final WeirdML v1 average-accuracy table. WeirdML v1 is excluded from BenchLM overall and category rankings.
44 of 44 rows
| Source rank | Source model or configuration | Average accuracy | Source setup | Parameters (B) | Open / closed |
|---|---|---|---|---|---|
| 1 | gemini-2.5-pro-exp-03-25Google | 61.05% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 2 | gpt-4.5-preview-2025-02-27OpenAI | 60.26% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 3 | o4-mini-2025-04-16 (high)OpenAI | 59.68% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 4 | o1-2024-12-17 (high)OpenAI | 59.47% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 5 | claude-3-7-sonnet-20250219 (thinking 8k)Anthropic | 55.69% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 6 | o3-2025-04-16 (high)OpenAI | 55.13% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 7 | gemini-2.5-flash-preview (thinking 16k)Google | 53.76% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 8 | Claude 3.7 SonnetAnthropic | 53.03% | Håvard Tveit IhleOriginal result | Not reported | Closed |
| 9 | GPT-4.1 miniOpenAI | 52.83% | Håvard Tveit IhleOriginal result | Not reported | Closed |
| 10 | Grok 3 [Beta]xAI | 52.08% | Håvard Tveit IhleOriginal result | Not reported | Closed |
| 11 | DeepSeek-R1DeepSeek | 51.44% | Håvard Tveit IhleOriginal result | Not reported | Open |
| 12 | o3-mini-2025-01-31 (high)OpenAI | 51.33% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 13 | Claude 3.5 SonnetAnthropic | 50.94% | Håvard Tveit IhleOriginal result | Not reported | Closed |
| 14 | DeepSeek-R1-0528DeepSeek | 50.37% | Håvard Tveit IhleOriginal result | Not reported | Open |
| 15 | deepseek-chat-v3-0324DeepSeek | 49.95% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 16 | GPT-4.1OpenAI | 49.20% | Håvard Tveit IhleOriginal result | Not reported | Closed |
| 17 | o1-previewOpenAI | 48.82% | Håvard Tveit IhleOriginal result | Not reported | Closed |
| 18 | grok-3-mini-beta (high)xAI | 47.67% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 19 | Qwen3 235B-A22BAlibaba | 45.87% | Håvard Tveit IhleOriginal result | Not reported | Open |
| 20 | o1-miniOpenAI | 45.58% | Håvard Tveit IhleOriginal result | Not reported | Closed |
| 21 | Llama 4 MaverickMeta | 44.24% | Håvard Tveit IhleOriginal result | Not reported | Open |
| 22 | gemini-2.0-flash-thinking-exp-01-21Google | 43.90% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 23 | Claude 3.5 HaikuAnthropic | 43.75% | Håvard Tveit IhleOriginal result | Not reported | Closed |
| 24 | qwen3-30b-a3bAlibaba | 41.06% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 25 | QwQ-32BAlibaba | 39.89% | Håvard Tveit IhleOriginal result | Not reported | Open |
| 26 | gemini-2.0-flash-expGoogle | 39.05% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 27 | gemini-2.0-pro-exp-02-05Google | 38.03% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 28 | DeepSeek V3DeepSeek | 37.37% | Håvard Tveit IhleOriginal result | Not reported | Open |
| 29 | GPT-4oOpenAI | 36.15% | Håvard Tveit IhleOriginal result | Not reported | Closed |
| 30 | GPT-4 TurboOpenAI | 33.60% | Håvard Tveit IhleOriginal result | Not reported | Closed |
| 31 | Mistral Medium 3Mistral | 31.92% | Håvard Tveit IhleOriginal result | Not reported | Closed |
| 32 | grok-2-1212xAI | 27.62% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 33 | qwq:32b-preview-q8_0Alibaba | 25.80% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 34 | GPT-4.1 nanoOpenAI | 25.73% | Håvard Tveit IhleOriginal result | Not reported | Closed |
| 35 | qwen2.5-coder:32b-instruct-q8_0Alibaba | 25.42% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 36 | GPT-4o miniOpenAI | 23.64% | Håvard Tveit IhleOriginal result | Not reported | Closed |
| 37 | llama3.3:70b-instruct-q8_0Meta | 23.48% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 38 | llama3.1:405b-instruct-q4_K_MMeta | 23.00% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 39 | phi4:14b-q8_0Microsoft | 13.16% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 40 | gemma-3-27b-itGoogle | 13.14% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 41 | Llama 4 ScoutMeta | 8.31% | Håvard Tveit IhleOriginal result | Not reported | Open |
| 42 | gemma2:27b-instruct-q8_0Google | 5.30% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 43 | gpt-3.5-turbo-0125OpenAI | 5.07% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
| 44 | llama3.1:8b-instruct-q8_0Meta | 1.96% | Håvard Tveit IhleOriginal result | Not reported | Unknown |
How to read this leaderboard
Higher is better. Because the score rewards reaching good results with fewer tokens, it is not comparable with WeirdML v2's average accuracy. Claude Fable 5.2 (xhigh, Claude Code) leads at 43.0, ahead of GPT-6 Astra (42.2) and GPT-6.1 Sol (38.3).
Operator receipt: 29 sourced rows are currently displayable on this page; the leading published row is Claude Fable 5.2 (xhigh) at 43.0%.
Honest limit: Rows are model plus harness plus, for open-weight models, an inference provider, so a score describes that setup. Runs per configuration are limited and the author publishes approximate 95% intervals. BenchLM shows WeirdML v3 as display-only; it does not enter overall or category rankings.
How WeirdML v3 is shown here
We mirror the official WeirdML v3 table from the author's prepared data file. WeirdML v3 gives coding agents 11 hand-made machine-learning and data-analysis tasks with unfamiliar data, unclear goals or little feedback, and scores how quickly and how well each agent improves as it spends tokens.
WeirdML v3 is display only on BenchLM. Each row is a model running inside a specific agent harness (Codex CLI, Claude Code, Gemini CLI, Kimi Code or OpenCode) and sometimes a specific inference provider, so BenchLM keeps the table separate from model-only rankings. The archived WeirdML v2 and v1 tables sit below it on this page; each version uses different tasks and scores, so compare within one table.
Snapshot
29 configurations11 tasks15 task configurationsOfficial data fileDisplay only
The published WeirdML v3 snapshot places Claude Fable 5.2 (xhigh) first at 43.0%. The third row is 4.7 points behind. The broader top-10 range is 24.0 points, so the table still separates the published systems.
29 models have been evaluated on WeirdML v3. The benchmark falls in the Coding category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. WeirdML v3 is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About WeirdML v3
Year
2026
Tasks
11 hand-made ML and data-analysis tasks (15 configurations)
Format
Agent harness runs up to 50M tokens, token-curve scoring
Difficulty
Open-ended ML engineering with unfamiliar data and little feedback
Each agent works on 11 tasks (15 task configurations, including hinted and hintless twins) inside its own harness, such as Codex CLI, Claude Code, Gemini CLI, Kimi Code or OpenCode. The official score is 80% of the normalized area under the agent's average best-so-far curve on a log token axis between 500K and 50M tokens, plus 20% of its final value; tasks count equally and hinted and hintless twins share a task's weight. Effective scores include normalization and hint penalties. Only models with at least one valid run in every configuration are listed. Epoch AI, METR and NDRE funded the API costs.
Freshness and provenance
Version
WeirdML v3 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
CurrentDisplay only
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
How is WeirdML v3 different from v2?
v3 uses 11 new, more open-ended tasks run inside agent harnesses for up to 50M tokens, and scores the whole improvement curve rather than final accuracy. Scores are on a different scale, so v2 and v3 numbers should not be compared.
Which model leads WeirdML v3?
Claude Fable 5.2 at xhigh effort in Claude Code leads with an official score of 43.0, followed by GPT-6 Astra in Codex CLI (42.2) and GPT-6.1 Sol (38.3).
Where are the WeirdML v2 and v1 results?
Both archived tables are on this page below the v3 table. WeirdML v2 has 162 models on 17 tasks; WeirdML v1 has 44 models on 6 tasks. Each version uses its own tasks, so compare within one table.
Last updated: WeirdML v3 · mirrored from the public benchmark leaderboard
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 7,000+ subscribers.
One email each week. Unsubscribe anytime.