BenchLM Research

LLM benchmark research & analysis

Evidence for choosing and evaluating AI models, grounded in benchmark explainers, model comparisons, and the BenchLM dataset.

68 published articles RSS feed

Two separate benchmark lanes labelled Terminal-Bench 2.0 and 2.1, with the 56.9 and 82.7 scores sitting in different lanes and the 61.8 like-for-like comparison marked in the 2.1 lane

Featured research

DeepSeek V4 Flash Gained 20.9 Points, Not 25.8

DeepSeek-V4-Flash-0731 shipped July 31 with MIT weights and $0.28 output pricing. The Terminal-Bench jump being quoted compares two benchmark versions, and DeepSeek's own table puts it behind Opus 4.8 on all nine rows.

Glevd · August 1, 2026 · 7 min read

Read the article

Latest research

A chronological record of benchmark analysis, model decisions, and evaluation practice.

Showing 67 of 68 articles

Jul 28, 2026Best LLM Observability ToolsAs of July 28, 2026, Langfuse is our pick for teams that need traces, evaluations, datasets, and an open-source self-hosting route in one product. LangSmith, Phoenix, Braintrust, and Helicone fit different operating constraints.Engineering guides13 min read

Jul 26, 2026Twelve Prompt Fixes, Including One RegressionWe ran twelve before-and-after AI prompt examples three times each. Revised prompts improved bounded work overall, but one detailed rewrite made the result worse.Research notes9 min read

Jul 26, 2026Celeris-1 Diffusion LLM for Faster Agentic AI WorkflowsCeleris-1 is a diffusion LLM built for low-latency agentic AI workflows. Review its provider-run speed tests, structured tasks, and OpenAI-compatible API.Sponsored research · Celeris9 min read

Jul 26, 2026Marketing Drafts That Stay Inside the FactsWe tested 24 marketing drafts against one product source ledger. The useful outcome was copy that followed the facts and requested format before editorial review.Research notes8 min read

Jul 26, 2026Write Prompts Around the Finished WorkA practical six-part method for writing AI prompts that produce testable work: outcome, context, sources, constraints, output contract, and acceptance check.Research notes9 min read

Jul 26, 2026Four Prompt Optimizers, One Test SetWe attempted 48 public prompt-optimizer rewrites and ran every usable artifact three times. Shared cases tied on accepted outputs; availability was the observed difference.Research notes8 min read

Jul 26, 2026Coding Changes That Survive ReviewWe ran 24 AI coding changes against a frozen repository. Better prompts did not improve the test-pass rate; they reduced unrelated code and review surface.Research notes9 min read

Jul 24, 2026The Latency Tax in Agentic WorkflowsAgentic workflows compound latency across model calls, tools, retries, and handoffs. Learn the core patterns, see practical examples, trace the critical path, and build a faster agentic AI workflow without sacrificing reliability.Research notes18 min read

Jul 20, 2026How to Optimize an AI PromptOptimize AI prompts with a test loop: preserve facts and constraints, define the output contract, test representative inputs, and fix one failure at a time.Research notes10 min read

Jul 17, 2026Is ChatGPT Plus Worth It?As of July 17, 2026, the $8 Go tier carries the same 160-messages-per-3-hours chat allowance as Plus, and OpenAI's own rate card budgets $100–$200 a month for a developer using Codex. We read every current OpenAI pricing page. Plus is worth $20 for exactly two kinds of user.Pricing and cost9 min read

Jul 16, 2026Kimi K3: The Open Model Closing the GapMoonshot released Kimi K3's full 2.8T-parameter weights on July 27, 2026. The open release is real; serving it still demands infrastructure on the scale of the model.Benchmark explainers5 min read

Jul 15, 2026Thinking Machines Chose Open Weights FirstWhy Thinking Machines made its first foundation-model release open weight, what Inkling changes for the lab, and where it falls short of the closed frontier.Benchmark explainers12 min read

Jul 14, 2026AI Coding Agents Need ReceiptsOn the July 14, 2026 SERP for 'ai coding agents', Vellum's list ranks Vellum first and Augment Code's list ranks Augment Code first. The one cross-agent table anyone publishes put Codex with GPT-5.6 Sol at 80 the same day. Here is the test we are running before we publish a ranking of our own.Benchmark explainers8 min read

Jul 11, 2026What Legal AI Benchmarks Actually MeasureFour legal AI benchmarks, four different leaders: Claude Fable 5 on LegalBench (88.6), GPT-5.6 Sol on Legal Research Bench (48.1), Muse Spark 1.1 on Harvey's Legal Agent Benchmark (20.0). The closer the test gets to real legal work, the lower every score gets.Benchmark explainers9 min read

Jul 8, 2026What Grok 4.5 Means for the FutureGrok 4.5 is not mainly a leaderboard story. At $2 input and $6 output per million tokens, it is a signal that closed frontier models are getting cheaper without becoming open source.Model comparisons10 min read

Jul 7, 2026ResearchClawBench: Why AI Science Agents Still Miss the PaperBenchLM added ResearchClawBench as a display-only benchmark. Claude Code leads at 21.5 RADS, far below the 50-point paper-reproduction line.Benchmark explainers15 min read

Jul 4, 2026HLE's 50% Ceiling Fell in 2026: What Broke It and What Is LeftFor two years, no AI model crossed 50% on Humanity's Last Exam. As of July 2026, fifteen models have, and the top tool-assisted score is 64.7. What broke the ceiling matters as much as the number.Model comparisons13 min read

Jul 4, 2026The Missing-Benchmark Problem: How Leaderboards Reward Hiding Weak ScoresA reader caught BenchLM ranking Qwen3.7 Max below its own cheaper sibling. The bug was not a data error. It was the averaging method almost every LLM leaderboard uses, and fixing it moved 170 scores.Benchmark explainers13 min read

Jul 4, 2026What AI Labs Don't Publish: The Benchmark Disclosure GapA July audit found far more published coding evidence than multilingual evidence among leading models. The disclosure gap still shapes rankings under BenchAlign v5.Benchmark explainers13 min read

Jul 2, 2026Introducing BenchLM Stats: Citable LLM Data, Regenerated on Every UpdateWe packaged the data behind BenchLM into 26 citable statistics across six pages — model prices, release cadence, context windows, benchmark saturation, open-source share, and market share. Every number is a self-contained, dated sentence generated from the live dataset, with a stable anchor URL you can cite.Research notes5 min read

Jul 2, 2026Best AI Web Scraping ToolsAs of July 28, 2026, Browse AI is our no-code shortlist, Firecrawl fits site-to-markdown jobs, and Apify offers the broadest developer platform of the three. We map documented fit and state plainly that we have not run a head-to-head extraction benchmark.Engineering guides9 min read

Jul 2, 2026Best Hosting Platforms for AI Apps in 2026: The Deploy LayerAs of July 2026, Netlify is our pick for short-running prototypes, Cloudflare for edge-scale production, and Railway or Fly.io for long-running backends. The limits that decide the choice.Engineering guides8 min read

Jul 2, 2026Best Text-to-Speech APIs for AI AppsAs of July 28, 2026, ElevenLabs is our default shortlist for a full voice stack, Cartesia for teams tuning a streaming path, and cloud TTS for existing cloud estates. The verdict is based on current product evidence, not an unrun listening benchmark.Engineering guides9 min read

Jul 2, 2026How to Deploy an AI App in 2026: From Model Pick to Production URLA practical guide to deploying AI apps and LLM-powered products — the model layer vs. the app layer, two runtime constraints, three operational checks, and the production pattern we use.Engineering guides15 min read

Jun 27, 2026Fable 5 vs GPT-5.6: Two Bets on Where the Frontier Goes NextAnthropic shipped Fable 5 and Mythos 5 at $10/$50. OpenAI moved GPT-5.6 Sol, Terra, and Luna to general availability on July 9 at a lower price ladder. The launch gap is the story of the 2026 AI market.Model comparisons16 min read

Jun 19, 2026How We Keep Pricing Data CurrentThe current pipeline starts with official first-party sources, a curated registry, validation, and human review. Downstream publication is automated; provider-page scraping is not.Engineering guides11 min read

Jun 12, 2026Best LLM for Voice AgentsThe right voice-agent model is the fastest one that clears your task, tool-use, and instruction-following tests. We use live first-answer data to build the shortlist, then show where that evidence stops.Model comparisons10 min read

Jun 12, 2026How to Get Cited by ChatGPT, Perplexity & Claude in 2026: AEO from the TrenchesA practitioner's guide to getting cited by ChatGPT, Perplexity, and Claude — the exact AEO/GEO changes we shipped on BenchLM: quotable lines, Dataset schema, llms.txt, AI-crawler access, and the tooling we use to find what to answer.Research notes12 min read

Jun 9, 2026Claude Fable 5 and Mythos 5: The Future of AI Is Gated IntelligenceAnthropic's Claude Fable 5 brings Mythos-class capability to public users, while Claude Mythos 5 remains trusted-access. The benchmark story is strong, but the real shift is capability-gated deployment.Benchmark explainers16 min read

May 12, 2026Best LLM for Math 2026: AIME, HMMT & MATH-500 RankingsBest LLM for math 2026: GPT-5.4 leads AIME 2025, MATH-500, and BRUMO. Compare Claude, Gemini, DeepSeek-R1, GPT-5.5, and value picks by use case.Model comparisons10 min read

May 12, 2026Perceptron Mk1 and Frontier Video Models: The Complete Guide to Video Understanding AIA complete guide to Perceptron Mk1, frontier video understanding models, video AI benchmarks, and where video-language models are headed next.Benchmark explainers52 min read

May 5, 2026ProgramBench Benchmark Explained: Can LLMs Rebuild Programs From Binaries?ProgramBench is a new LLM coding benchmark where agents rebuild full programs from a compiled binary and documentation. See scores, how it differs from SWE-bench, and why all public models are 0% resolved.Benchmark explainers17 min read

Apr 27, 2026ARC-AGI-2 Explained: The Hardest Public Reasoning BenchmarkARC-AGI-2 measures fluid intelligence through visual grid puzzles that can't be solved by memorization. Here's how it works, what scores mean, and where current frontier models stand.Benchmark explainers17 min read

Apr 24, 2026Advertised vs Effective Context Windows: What 1M-Token Claims HideFour frontier LLMs advertise 1M+ tokens, but advertised and effective context are different numbers. DeepSeek V4 Pro's 384K output changes generation workflows; Gemini leads effective-context evals. What the headline claims hide.Model comparisons14 min read

Apr 24, 2026DeepSeek V4 Pro vs Claude Opus 4.7 vs GPT-5.5: The Frontier in April 2026Three frontier flagships launched in eight days. Here is how DeepSeek V4 Pro, GPT-5.5, and Claude Opus 4.7 compare on benchmarks, current cost, and real use.Model comparisons17 min read

Apr 9, 2026GPT-5 vs Gemini in 2026: Full Benchmark BreakdownGPT-5.6 Sol vs Gemini 3.5 Flash on current BenchAlign overall, coding, agentic, price, context, and evidence data.Model comparisons8 min read

Apr 7, 2026Mythos Preview is the first frontier model Anthropic decided not to ship. The benchmarks show why.Claude Mythos Preview beats Opus 4.6 by double digits on every coding benchmark Anthropic released. Then they shelved it. Here's what the numbers actually show, and why the shipping decision matters more than the launch.Benchmark explainers12 min read

Apr 6, 2026Choosing the Best LLM for RAGAs of July 28, 2026, we do not publish a synthetic RAG ranking. The best model is the cheapest candidate that clears your grounded-answer eval; live long-context and instruction-following rows build the shortlist.Model comparisons11 min read

Apr 6, 2026Best LLM for Writing (July 2026): Fable 5 vs GPT-5.6 TestedWhich AI model is best for writing? Claude Fable 5 holds the top Arena Elo (1508) on BenchLM's board, with GPT-5.6 Sol the new challenger. Claude, GPT, and Gemini ranked by creative-writing and instruction-following scores, with pricing for every budget.Model comparisons10 min read

Apr 4, 2026How to Choose an LLM in 2026: Best by Use CaseChoose the right LLM using current overall, coding, agentic, open-weight, price, access, and evidence data instead of one universal winner.Model comparisons9 min read

Apr 1, 2026Best Open-Weight LLM: MiniMax M3 LeadsMiniMax M3 leads the July 2026 open-weight ranking, followed by GLM-5.1 and GLM-5. Scores, evidence strength, coding trade-offs, and self-hosting choices.Model comparisons12 min read

Mar 30, 2026Why Chinese LLM Rankings DisagreeThe provisional lane picked Qwen3.7 Max; BenchAlign v5 picked MiMo-V2.5-Pro. A dated audit of the contract, identity, and creator-filter split.Model comparisons7 min read

Mar 30, 2026ChatGPT vs Claude vs Gemini in 2026: Which One Should You Use?ChatGPT, Claude, and Gemini compared with current BenchAlign scores, coding and agentic rankings, prices, evidence strength, and practical use cases.Model comparisons8 min read

Mar 26, 2026How LLM Token Pricing Works: A Complete Guide to API Costs in 2026Learn how LLM API pricing works — from tokens, input/output costs, and reasoning tokens to vision, embedding, and fine-tuning pricing. Includes real cost examples, free tiers, and 6 strategies to cut your AI spend.Pricing and cost18 min read

Mar 24, 2026React Native Evals: The Mobile App Coding Benchmark ExplainedReact Native Evals measures whether AI coding models can complete real React Native implementation tasks across navigation, animation, and async state. Here's what it tests, why it matters, and how it differs from SWE-bench and LiveCodeBench.Benchmark explainers7 min read

Mar 22, 2026State of LLM Benchmarks (July 2026): 296 Evals TrackedThe July 2026 state of LLM benchmarks: 296 tracked evaluations, current overall, coding, agentic, and open-weight leaders, plus what BenchAlign changes about missing data.Model comparisons10 min read

Mar 18, 2026Are AI Benchmarks Reliable? The Data Contamination ProblemAI benchmarks are useful but flawed. Data contamination inflates scores when models train on test questions. Here's how it works, which benchmarks resist it, and how BenchLM accounts for reliability.Benchmark explainers9 min read

Mar 18, 2026Best Budget LLMs in 2026: GPT-5.4 Mini, Nano, MiniMax M2.7, and Every Cheap Model RankedWhich budget LLM should you use in 2026? We rank GPT-5.4 mini, GPT-5.4 nano, MiniMax M2.7, Claude Haiku 4.5, Gemini Flash, DeepSeek, and more by benchmarks and price.Model comparisons12 min read

Mar 12, 2026Best LLM for Coding: Mythos Leads, Fable ShipsClaude Mythos 5 leads the July 2026 coding ranking, Claude Fable 5 is the strongest generally available pick, and evidence labels show where sparse rows remain uncertain.Model comparisons9 min read

Mar 12, 2026BrowseComp Explained: How We Measure Web Research AgentsBrowseComp evaluates whether AI models can search the web, gather evidence, and answer research questions instead of relying only on latent knowledge.Benchmark explainers6 min read

Mar 12, 2026Claude Opus 4.6 vs GPT-5.4: Full Benchmark Breakdown (2026)Claude Opus 4.6 vs GPT-5.4 with current BenchAlign scores, coding and agentic evidence, raw benchmarks, pricing, and the newer models you should also consider.Model comparisons8 min read

Mar 12, 2026LLM Pricing 2026: Every Model, $0.11–$50 per 1M TokensHow to actually compare LLM API pricing — what blended cost means, which discounts matter (caching, batch), and how to find the cheapest model for your workload. With current rates for GPT-5, Claude, Gemini, DeepSeek, and more.Model comparisons8 min read

Mar 12, 2026OSWorld-Verified Explained: How We Measure Computer-Use ModelsHow OSWorld-Verified tests computer-use systems, how its sourced results should be compared, and why OSWorld 2.0 is a separate protocol.Benchmark explainers7 min read

Mar 12, 2026Terminal-Bench 2.0 Explained: How We Measure Agentic CodingTerminal-Bench 2.0 measures whether AI models can work through real terminal-based coding and ops workflows instead of just answering in chat.Benchmark explainers6 min read

Mar 12, 2026What Do LLM Benchmarks Actually Measure?LLM benchmarks don't measure intelligence. They measure specific, narrow abilities under controlled conditions. Here's what each benchmark type actually tests — and what it misses.Benchmark explainers10 min read

Mar 7, 2026AIME & HMMT: Can AI Models Do Competition Math?AIME and HMMT are high school math olympiad competitions now used to benchmark AI. Frontier models score 95-99% — competition math is effectively solved. Here's what that means.Benchmark explainers10 min read

Mar 7, 2026Arena Elo Explained: How LMArena Chatbot Rankings WorkArena (formerly Chatbot Arena / LMArena) ranks AI models with Elo ratings from blind human preference votes. How the system works, what scores mean, and how Elo compares to benchmarks.Benchmark explainers10 min read

Mar 7, 2026Claude Opus 4.6 vs GPT-5.4: Where Each Model WinsClaude Opus 4.6 vs GPT-5.4 on current overall, coding, agentic, raw benchmark, and pricing data, with the evidence caveats that decide the close calls.Model comparisons6 min read

Mar 7, 2026GPQA Diamond: The PhD-Level Science BenchmarkGPQA tests AI models with graduate-level questions in biology, physics, and chemistry that are 'Google-proof' — even skilled non-experts with internet access can't answer them. Here's how it works.Benchmark explainers10 min read

Mar 7, 2026HLE (Humanity's Last Exam): The Hardest BenchmarkHumanity's Last Exam is crowdsourced from thousands of domain experts and designed to probe the absolute frontier of AI. Top models still top out under 65%. Here's why HLE matters.Benchmark explainers10 min read

Mar 7, 2026LiveCodeBench: Why Static Coding Benchmarks Aren't EnoughHow LiveCodeBench uses fresh contest problems, why release and metric labels matter, and what its scores do not say about repository engineering.Benchmark explainers8 min read

Mar 7, 2026MMLU vs MMLU-Pro: What Changed and Why It MattersMMLU and MMLU-Pro are the most cited knowledge benchmarks in AI. Here's what each measures, why MMLU is saturated, and why MMLU-Pro is the better discriminator in 2026.Benchmark explainers7 min read

Mar 7, 2026SWE-bench Explained: How We Measure Real-World CodingSWE-bench Verified tests AI models on resolving real GitHub issues from Django, Flask, and scikit-learn. Here's how it works, why it matters, and which models score highest.Benchmark explainers7 min read

Mar 7, 2026What Is HumanEval? The Coding Benchmark ExplainedHumanEval tests whether AI models can generate correct Python functions from docstrings. Here's what it measures, why it's nearly saturated, and which benchmarks matter more in 2026.Benchmark explainers6 min read

Aug 22, 2025Building Your Own LLM Benchmark: A Practical GuideHow to create a custom LLM benchmark for your specific use case — from defining tasks and building datasets to scoring models and avoiding common pitfalls.Benchmark explainers12 min read

Aug 22, 2025The Complete Guide to LLM Benchmarking: Everything You Need to KnowEverything you need to know about LLM benchmarking — what benchmarks measure, how to choose the right ones, common pitfalls, and how to interpret results for real-world model selection.Benchmark explainers15 min read

Aug 22, 2025How to Interpret LLM Benchmark Results: A Practical GuideHow to read LLM benchmark scores correctly — what differences are meaningful, what to ignore, common misinterpretations, and how to translate benchmark data into model selection decisions.Benchmark explainers10 min read

Get the BenchLM weekly brief

New analysis, material benchmark and price changes, and one model decision worth revisiting.

Read a sample issue

Join 2,000+ readers.