Vals Inference Provider Index v1 (Vals Inference Provider Index)
We show this table for reference; we do not rank on it.
Vals AI comparison of the same open-weight model served by different inference providers, against the model creator's own API, on agentic finance, coding and legal tasks.
Evaluation results published by Vals AI. Published leaderboard aggregates and evaluation settings only. Dataset contents are not included.
Accuracy on Vals Inference Provider Index — October 20, 2026
We mirror the published accuracy view for Vals Inference Provider Index. Kimi K3 · Fireworks leads the public snapshot at 73.85%, followed by Kimi K3 · Together AI (73.18%) and DeepSeek V4.1 Flash · Fireworks (72.99%). We do not use these results to rank models overall.
Kimi K3 · Fireworks
Fireworks AI
Kimi K3 · Together AI
Together AI
DeepSeek V4.1 Flash · Fireworks
Fireworks AI
11 modelsAgenticCurrentDisplay onlyUpdated October 20, 2026
Accuracy table (11 models)
| Rank | Model / configuration | Score | Parameters (B) | Open / closed |
|---|---|---|---|---|
| 1 | Kimi K3 · FireworksFireworks AI | 73.85% | Not reported | Unknown |
| 2 | Kimi K3 · Together AITogether AI | 73.18% | Not reported | Unknown |
| 3 | DeepSeek V4.1 Flash · FireworksFireworks AI | 72.99% | Not reported | Unknown |
| 4 | Kimi K3 · Moonshot APIMoonshot AImax reasoning | 71.77% | Not reported | Unknown |
| 5 | DeepSeek V4.1 Flash · DeepSeek APIDeepSeekhigh reasoning | 70.33% | Not reported | Unknown |
| 6 | DeepSeek V4.1 Flash · BasetenBaseten | 68.86% | Not reported | Unknown |
| 7 | Kimi K3 · BasetenBaseten | 67.28% | Not reported | Unknown |
| 8 | DeepSeek V4.1 Flash · Together AITogether AI | 66.39% | Not reported | Unknown |
| 9 | MiniMax M3 · MiniMax APIMiniMax | 42.88% | Not reported | Unknown |
| 10 | MiniMax M3 · FireworksFireworks AI | 39.35% | Not reported | Unknown |
| 11 | MiniMax M3 · Together AITogether AI | 37.63% | Not reported | Unknown |
How to read this leaderboard
Operator receipt: 11 sourced rows are currently displayable on this page; the leading published row is Kimi K3 · Fireworks at 73.85%.
Honest limit: Three models and four providers only, each measured at one point in time; providers change their serving stacks behind the same model name. Vals reports most provider gaps as within error. Rows describe a model served by one provider, not a different model.
How Vals Inference Provider Index is shown here
We mirror Vals AI's Inference Provider Index, updated October 20, 2026. It runs DeepSeek V4.1 Flash, Kimi K3 and MiniMax M3 on their creators' APIs and on Together AI, Fireworks and Baseten, using the same agent and 165 tasks from Finance Agent v2, Vibe Code Bench and Legal Research. MiniMax M3 was not available on Baseten.
Each row is one model served by one provider, so the board compares serving stacks rather than models. Vals reports accuracy, cost per task and speed, and reruns the index every two weeks. The results stay display only and do not enter model rankings.
Snapshot
The published Vals Inference Provider Index snapshot places Kimi K3 · Fireworks first at 73.85%. The third row is 0.86 points behind. The broader top-10 range is 34.51 points, so the table still separates the published systems.
11 models have been evaluated on Vals Inference Provider Index. The benchmark falls in the Agentic category. We keep external benchmark mirrors separate from the weighted global scoring system, so these results remain source-specific evidence. Vals Inference Provider Index is currently displayed for reference but excluded from the scoring formula, so it does not directly affect overall rankings.
About Vals Inference Provider Index
Year
2026
Tasks
165 agentic tasks from three Vals Index benchmarks
Format
Accuracy, cost per task and speed by model and provider
Difficulty
Long tool-using agent work in finance, coding and law
Vals published v1 on October 20, 2026 and reruns it every two weeks. DeepSeek V4.1 Flash, Kimi K3 and MiniMax M3 each run on their creator's API and on Together AI, Fireworks and Baseten (MiniMax M3 was not available on Baseten) over a fixed 165-task set: 66 Finance Agent v2, 33 Vibe Code Bench and 66 Legal Research tasks, with the same agent and aligned sampling settings. Vals reports accuracy, cost per task and speed. We mirror the published aggregates as a display-only board, outside overall and category rankings.
Freshness and provenance
Version
Vals Inference Provider Index 2026
Refresh cadence
Quarterly
Staleness state
Current
Question availability
Public benchmark set
BenchLM uses freshness metadata to decide whether a benchmark should still be treated as a strong differentiator, a benchmark to watch, or a display-only reference. For the full scoring policy, see the BenchLM methodology page.
Questions
What does Vals Inference Provider Index measure?
Vals AI comparison of the same open-weight model served by different inference providers, against the model creator's own API, on agentic finance, coding and legal tasks.
Which model leads the published Vals Inference Provider Index snapshot?
Kimi K3 · Fireworks currently leads the published Vals Inference Provider Index snapshot with 73.85% accuracy. BenchLM shows this benchmark for display only and does not use it in overall rankings.
How many models are evaluated on Vals Inference Provider Index?
The October 20, 2026 snapshot contains 11 AI models.
Know when it’s worth switching models
The model to choose, the cheaper alternative, and the release we would wait on.
Read a sample issueJoin 6,100+ subscribers.
One email each week. Unsubscribe anytime.