SciCode Main Standard (Kaggle task v3) (SciCode Main Standard (Kaggle))
We show this table for reference; we do not rank on it.
A separate Kaggle SciCode track for main research problems without background notes.
Evaluation results published by Kaggle Benchmarks. How Kaggle Benchmarks reruns tests.
Problem success rate on SciCode Main Standard (Kaggle) — 2026-10-20 source capture
We mirror the published problem success rate view for SciCode Main Standard (Kaggle). Claude Opus 5.5 has the highest reported value at 18.5%. These are published source configurations. Compare the task version and disclosed settings before treating a score difference as a model difference. We do not use these results to rank models overall.
Claude Opus 5.5
Task version 3; Evaluated 2026-09-24
Gemini 3.1 Pro Preview
Task version 3
Claude Opus 5
Task version 3; Evaluated 2026-07-27
62 configurationsCodingRefreshingDisplay onlyUpdated 2026-10-20 source capture
Problem success rate table (62 configurations)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Claude Opus 5.5Creator not reportedTask version 3; Evaluated 2026-09-24 | 18.5% | Not reported | Unknown |
| 2 | Gemini 3.1 Pro PreviewCreator not reportedTask version 3 | 14.3% | Not reported | Unknown |
| 3 | Claude Opus 5Creator not reportedTask version 3; Evaluated 2026-07-27 | 13.8% | Not reported | Unknown |
| 4 | Gemini 3.7 FlashCreator not reportedTask version 3; Evaluated 2026-08-28 | 13.8% | Not reported | Unknown |
| 5 | Claude Sonnet 5.5Creator not reportedTask version 3; Evaluated 2026-09-30 | 13.8% | Not reported | Unknown |
| 6 | Gemini 3 Flash PreviewCreator not reportedTask version 3 | 12.7% | Not reported | Unknown |
| 7 | Claude Opus 4.8Creator not reportedTask version 3 | 12.3% | Not reported | Unknown |
| 8 | Gemini 3.8 FlashCreator not reportedTask version 3; Evaluated 2026-09-06 | 12.3% | Not reported | Unknown |
| 9 | GPT-6 AstraCreator not reportedTask version 3; Evaluated 2026-09-14 | 12.3% | Not reported | Unknown |
| 10 | Gemini 4 ArgonCreator not reportedTask version 3; Evaluated 2026-10-12 | 12.3% | Not reported | Unknown |
| 11 | Gemini 3.5 FlashCreator not reportedTask version 3 | 12.3% | Not reported | Unknown |
| 12 | o4 miniCreator not reportedTask version 3; Evaluated 2025-05-22 | 10.8% | Not reported | Unknown |
| 13 | GPT-5.6 SolCreator not reportedTask version 3; Evaluated 2026-07-22 | 10.8% | Not reported | Unknown |
| 14 | GPT-5.6 TerraCreator not reportedTask version 3; Evaluated 2026-07-22 | 10.8% | Not reported | Unknown |
| 15 | Gemini 3.6 FlashCreator not reportedTask version 3; Evaluated 2026-07-21 | 10.8% | Not reported | Unknown |
| 16 | Claude Sonnet 5Creator not reportedTask version 3; Evaluated 2026-07-23 | 10.7% | Not reported | Unknown |
| 17 | GPT-6.1 SolCreator not reportedTask version 3; Evaluated 2026-10-05 | 9.2% | Not reported | Unknown |
| 18 | Claude Haiku 5.5Creator not reportedTask version 3; Evaluated 2026-10-09 | 9.2% | Not reported | Unknown |
| 19 | o3Creator not reportedTask version 3; Evaluated 2025-05-22 | 9.2% | Not reported | Unknown |
| 20 | Gemma 4 31BCreator not reportedTask version 3 | 8.9% | Not reported | Unknown |
| 21 | o3 miniCreator not reportedTask version 3; Evaluated 2025-05-22 | 7.7% | Not reported | Unknown |
| 22 | Claude Sonnet 4Creator not reportedTask version 3; Evaluated 2025-05-23 | 7.7% | Not reported | Unknown |
| 23 | Gemini 2.5 Pro PreviewCreator not reportedTask version 3; Evaluated 2025-06-06 | 7.7% | Not reported | Unknown |
| 24 | gpt-oss-120bCreator not reportedTask version 3; Evaluated 2025-08-15 | 7.7% | Not reported | Unknown |
| 25 | Grok 4.5Creator not reportedTask version 3; Evaluated 2026-07-21 | 6.5% | Not reported | Unknown |
| 26 | GPT-5.6 LunaCreator not reportedTask version 3; Evaluated 2026-07-21 | 6.2% | Not reported | Unknown |
| 27 | Gemini 3.5 Flash-LiteCreator not reportedTask version 3; Evaluated 2026-07-21 | 6.2% | Not reported | Unknown |
| 28 | Claude Opus 4Creator not reportedTask version 3; Evaluated 2025-05-23 | 6.2% | Not reported | Unknown |
| 29 | Gemini 1.5 ProCreator not reportedTask version 3; Evaluated 2025-05-22 | 4.6% | Not reported | Unknown |
| 30 | o1Creator not reportedTask version 3; Evaluated 2025-05-22 | 4.6% | Not reported | Unknown |
| 31 | Claude 3.7 SonnetCreator not reportedTask version 3; Evaluated 2025-05-22 | 4.6% | Not reported | Unknown |
| 32 | GPT-4.1Creator not reportedTask version 3; Evaluated 2025-05-22 | 4.6% | Not reported | Unknown |
| 33 | DeepSeek-R1Creator not reportedTask version 3; Evaluated 2025-06-04 | 4.6% | Not reported | Unknown |
| 34 | Qwen 3 235B A22B Instruct 2506Creator not reportedTask version 3; Evaluated 2025-09-03 | 4.6% | Not reported | Unknown |
| 35 | Claude Opus 4.1Creator not reportedTask version 3; Evaluated 2025-08-26 | 4.6% | Not reported | Unknown |
| 36 | Gemini 3.1 Flash-Lite PreviewCreator not reportedTask version 3 | 4.6% | Not reported | Unknown |
| 37 | o1 miniCreator not reportedTask version 3; Evaluated 2025-05-22 | 3.1% | Not reported | Unknown |
| 38 | DeepSeek-V3Creator not reportedTask version 3; Evaluated 2025-05-22 | 3.1% | Not reported | Unknown |
| 39 | Gemini 2.5 Flash PreviewCreator not reportedTask version 3; Evaluated 2025-05-23 | 3.1% | Not reported | Unknown |
| 40 | deepseek-v3.1Creator not reportedTask version 3; Evaluated 2025-08-27 | 1.5% | Not reported | Unknown |
| 41 | Claude 3.5 SonnetCreator not reportedTask version 3; Evaluated 2025-05-22 | 1.5% | Not reported | Unknown |
| 42 | GPT-4oCreator not reportedTask version 3; Evaluated 2025-05-22 | 1.5% | Not reported | Unknown |
| 43 | Gemini 2.0 FlashCreator not reportedTask version 3; Evaluated 2025-05-22 | 1.5% | Not reported | Unknown |
| 44 | GPT-4o miniCreator not reportedTask version 3; Evaluated 2025-05-22 | 1.5% | Not reported | Unknown |
| 45 | Claude 3.5 HaikuCreator not reportedTask version 3; Evaluated 2025-05-22 | 1.5% | Not reported | Unknown |
| 46 | Mistral Large 2Creator not reportedTask version 3; Evaluated 2025-05-22 | 1.5% | Not reported | Unknown |
| 47 | Grok 3Creator not reportedTask version 3; Evaluated 2025-05-26 | 1.5% | Not reported | Unknown |
| 48 | gpt-oss-20bCreator not reportedTask version 3; Evaluated 2025-08-15 | 1.5% | Not reported | Unknown |
| 49 | Gemini 1.5 FlashCreator not reportedTask version 3; Evaluated 2025-05-22 | 0.0% | Not reported | Unknown |
| 50 | Gemini 1.5 Flash 8BCreator not reportedTask version 3; Evaluated 2025-05-22 | 0.0% | Not reported | Unknown |
| 51 | Grok 2Creator not reportedTask version 3; Evaluated 2025-05-22 | 0.0% | Not reported | Unknown |
| 52 | GPT-3.5 TurboCreator not reportedTask version 3; Evaluated 2025-05-22 | 0.0% | Not reported | Unknown |
| 53 | Ministral 3BCreator not reportedTask version 3; Evaluated 2025-05-22 | 0.0% | Not reported | Unknown |
| 54 | Ministral 8BCreator not reportedTask version 3; Evaluated 2025-05-22 | 0.0% | Not reported | Unknown |
| 55 | CodestralCreator not reportedTask version 3; Evaluated 2025-05-22 | 0.0% | Not reported | Unknown |
| 56 | Mixtral 8x22BCreator not reportedTask version 3; Evaluated 2025-05-22 | 0.0% | Not reported | Unknown |
| 57 | Gemma 3 4BCreator not reportedTask version 3; Evaluated 2025-05-22 | 0.0% | Not reported | Unknown |
| 58 | Gemma 3 27BCreator not reportedTask version 3; Evaluated 2025-05-22 | 0.0% | Not reported | Unknown |
| 59 | Gemma 3 12BCreator not reportedTask version 3; Evaluated 2025-05-22 | 0.0% | Not reported | Unknown |
| 60 | Grok 3 MiniCreator not reportedTask version 3; Evaluated 2025-05-26 | 0.0% | Not reported | Unknown |
| 61 | GPT-5Creator not reportedTask version 3 | 0.0% | Not reported | Unknown |
| 62 | Granite 4.0 SmallCreator not reportedTask version 3; Evaluated 2025-09-30 | 0.0% | Not reported | Unknown |
How to read this leaderboard
Operator receipt: 62 sourced rows are currently displayable on this page; the leading published row is Claude Opus 5.5 at 18.5%.
Honest limit: Background notes change the knowledge available in the prompt. Main-problem and subproblem results are different tasks. Source intervals have no disclosed confidence level in this API response, and missing settings are not inferred.
How to read these results
This Kaggle SciCode track tests main research problems without background notes. The four tracks retain separate scores because supplying background information and evaluating subproblems change the task.
We preserve Kaggle task version 3, source configuration names, evaluation dates, and published intervals. The source does not state an interval confidence level here. These rows remain separate from the core SciCode, Artificial Analysis, and Mercor evaluation protocols.
Snapshot
These are published source configurations. Compare the task version and disclosed settings before treating a score difference as a model difference. The table retains each source configuration and its reported value.
62 configurations are shown for SciCode Main Standard (Kaggle). The benchmark falls in the Coding category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. SciCode Main Standard (Kaggle) is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About SciCode Main Standard (Kaggle)
Year
2024
Tasks
Scientific research main problems
Format
Published Kaggle leaderboard aggregates
This track reports problem success for main research problems without background notes. We show the published percentage and keep it separate from the other three Kaggle tracks and other SciCode evaluation providers.
Freshness and provenance
Version
SciCode Main Standard (Kaggle) 2024
Refresh cadence
Annual
Staleness state
Refreshing
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does SciCode Main Standard (Kaggle) measure?
A separate Kaggle SciCode track for main research problems without background notes.
Which source row has the highest reported SciCode Main Standard (Kaggle) value?
Claude Opus 5.5 has the highest reported SciCode Main Standard (Kaggle) value at 18.5%. These are published source configurations. Compare the task version and disclosed settings before treating a score difference as a model difference. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on SciCode Main Standard (Kaggle)?
The 2026-10-20 source capture snapshot contains 62 AI models.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 6,100+ subscribers.
One email each week. Unsubscribe anytime.