Research deskBenchLM Research

107 published articles RSS feed

LLM benchmark research & analysis

Evidence for choosing and evaluating AI models, grounded in benchmark explainers, model comparisons, and the BenchLM dataset.

Five hand-inked discs inside a thin pencil corner on used sage ledger paper: one low at the left, a deep cobalt stain in the middle, a filled disc above a hollow ring, and one disc at the upper right

Featured research

GPT-6 models compared: Astra, Sol, Terra and Luna

OpenAI's GPT-6 family has five API models: GPT-6 Astra, GPT-6.1 Sol, GPT-6 Sol, GPT-6.1 Terra and GPT-6 Luna. GPT-6.1 Sol replaces GPT-6 Sol at the same price, and GPT-6.1 Terra comes close to it for half. Prices, scores and plan access as of October 29, 2026.

Glevd · October 29, 2026 · 6 min read

Read the article

Latest research

A chronological record of benchmark analysis, model decisions, and evaluation practice.

Showing 106 of 107 articles, plus the featured article above

Twenty hand-inked marks on ash ledger paper, one per model in the October 29 overall top 20, stepping down by score: a hollow ring at thirteenth for Kimi K3.1 Pro, whose weights are pending, and a cobalt stain at seventeenth for Kimi K3, the best open-weight model

Oct 29, 2026What open weights let you do, and what they don'tAn open-weight model publishes its trained weights for anyone to download, run, and fine-tune. As of October 29, 2026, the best one ranks 17th on our board, most leaders need a datacenter, and the license decides the rest.Benchmark explainers13 min read

One inked tank split by a single line beside two separate tanks, one of them pooled with cobalt, on a stone ledger sheet

Oct 21, 2026Claude Duet starts at $20 a month, Dots at $100Claude Duet comes with Claude Pro and Max but spends the same limits as your other Claude work. OpenAI Dots needs ChatGPT Pro or Business Premium, and conversations with it don't count against ChatGPT limits. Compare usage, approvals, visibility, and regions as of October 21, 2026.Pricing and cost6 min read

A row of small inked boxes, each with its own tick, beside a single long bracket over a matching row and a cobalt stain, on a sand ledger sheet

Oct 21, 2026Claude Duet and Muse draw the approval line differentlyClaude Duet asks before every send, payment, signature, or deletion and shares your Claude plan limits. Meta Muse lets standing grants cover later actions and meters a separate weekly allowance. Compare approvals, usage, and access as of October 21, 2026.Pricing and cost6 min read

Three hand-inked bars of rising length on used ochre ledger paper, the third eight times the first, and below them a shorter cobalt bar struck through with one ink stroke

Oct 13, 2026Video generation API costs by resolution, tier, and routeVeo 3.1 per-second prices by tier and resolution, worked 8-second budgets, and the shutdown dates for the Gemini API, Vertex AI, OpenAI, and Azure routes.Research notes8 min read

Three hand-inked rows on used rust ledger paper, each a short stroke at the same October column joined by a thin line to a second stroke, the top row reaching farthest to a cobalt stroke and stain

Oct 13, 2026GPT-4o, o1, and o3-mini on OpenAI's October 23 shutdown listOpenAI shuts down gpt-4o-2024-05-13, o1, and o3-mini on October 23. Azure's dates are December 9 and November 19. Inventory, test, and switch with records.Research notes8 min read

Two uneven inked door latches, one seated and one pulled back beside a cobalt stain, on a sage ledger sheet

Oct 6, 2026Choosing Hark Pro or Muse by connected accountsHark Pro offers custom Panels and suggested Action Buttons. Meta Muse connects to WhatsApp and business tools. Compare access, published usage, and approval settings before choosing.Pricing and cost6 min read

A hand-inked measuring jug on used putty ledger paper, filled at the mouth with a large cobalt ink stain and no ink trail from its spout

Oct 5, 2026Reasoning tokens count toward the bill and capThinking uses billed tokens before an answer appears. Check current API usage fields, a hypothetical cost example, and the limits of five archived records.Pricing and cost10 min read

Two hand-inked entry gates on sage ledger paper, meeting at a cobalt stain

Sep 30, 2026Choosing Dots or Grok Bot starts with your accountCompare OpenAI Dots and Grok Bot by account eligibility, cloud and local computers, model control, permissions, and how included usage becomes extra charges.Pricing and cost8 min read

An open gate drawn in wet black ink on slate ledger paper, with a cobalt pool across the threshold

Sep 30, 2026What your Dot can access and doOpenAI Dots in ChatGPT: current Pro and Business Premium prices, regional access, setup steps, permissions, and how Codex and Work tasks use your allowance.Pricing and cost9 min read

A black ink receipt curl on ochre ledger paper, with a cobalt ink spill beneath its lower edge

Sep 30, 2026What it costs to run OpenClawOpenClaw software costs $0. Estimate model, tool, hosting, storage, and operator costs with a blank worksheet that keeps unknown charges visible.Pricing and cost8 min read

Two hand-inked keys on ledger paper, with cobalt on the left key's round head

Sep 30, 2026Choosing between OpenClaw and Hermes AgentCompare OpenClaw and Hermes Agent by setup, model access, hosting, permissions, and billing. Choose a configuration before judging its task results.Pricing and cost10 min read

OpenAI DevDay 2026 title on a black background, with DevDay in green and 2026 in purple

Sep 29, 2026What OpenAI announced at DevDayOpenAI DevDay 2026 introduced Dots, ChatGPT Space, GPT-6.1 Sol, Ultrafast, new agent APIs, and new ways to distribute software. Launches, previews, prices, and limits.Pricing and cost13 min read

Two rows of ink tally strokes on rust-red ledger paper, one stroke per OpenAI model page: the lower row is complete but for one gap, the upper row has nine gaps, and one stroke in the upper row is cobalt

Sep 25, 2026Chat Completions still works. Your tools may not.Chat Completions is still supported, but GPT-6 Astra's tools need Responses. Dated API status, every parameter change, silent breaks and a 540-call test.Research notes11 min read

Eighty-seven ink tally strokes on ochre ledger paper, one per API call: the first is a tall cobalt stroke rising above a dashed pencil line, and the other eighty-six are short black ticks along the bottom

Sep 25, 2026Prompt caching pays on the second callPrompt caching bills a repeated prompt prefix at a fraction of the input price, and some providers charge to write it. Rules, prices and 174 logged API calls.Pricing and cost12 min read

Two hand-drawn test paths on slate ledger paper meet at one cobalt-marked failed row

Sep 22, 2026The evaluator must expose the failureCompare two LLM evaluation frameworks, Promptfoo and DeepEval, on one 372-row saved-output replay, then choose by stack, failure inspection, and maintenance.Research notes13 min read

One ink stroke that forks into three branches on teal ledger paper, the lower branch ending in a cobalt blot

Sep 22, 2026Jev cannot write. That is the feature and the limit.TypeSafe's Jev returns typed decisions instead of prose. How it works, what its confidence means, where it fails, and what our LLM Selector sends to it.Research notes12 min read

A hand-inked meter bar on ochre ledger paper, ink hatching drained to the right, a cobalt end segment and a cobalt pool below

Sep 17, 2026API credits expire, cap you and differ by providerAPI credits are money you pay a provider before you use its models, drawn down per token. OpenAI, Anthropic and Google each sell them on different terms: minimums, expiry after a year, refund rules, auto-recharge and the usage tiers your payments unlock.Research notes9 min read

An ink prompt box on fog ledger paper, an arrow, and nine token squares trailing right, the last one cobalt

Sep 17, 2026Inference is the part of AI you pay forInference is the moment a trained model answers a prompt. Training built the weights once. Inference runs them on every request, is priced per token, and is where latency, throughput and your bill all come from.Research notes9 min read

Two ink checkmarks on ivory ledger paper, the second crossed by a single diagonal cobalt strike

Sep 14, 2026Valid JSON can still be wrongStructured output guarantees the shape of a response, not the truth in it. Four synthetic pairs from one support-ticket fixture show a schema pass hiding a wrong field, then two filed runs of Claude Sonnet 5 and GPT-5.6 Terra show the same failure class with denominators.Research notes13 min read

Sep 11, 2026Mercury 2.5 upgrades the calls you don't seeInception's new model provides a 40% intelligence gain over Mercury 2 while running at 1,107 tokens per second, with launch pricing at just $0.04 per million input tokens. The interesting question is what that does to the steps in your pipeline that run ten thousand times a day.Sponsored research · Inception5 min read

A thick redacted ink bar on ash ledger paper, its right half crossed by a diagonal cobalt strike with a cobalt pool at the pivot

Sep 4, 2026Deprecated means flagged, not brokenDeprecated means still working, stop building on it. Five AI providers spell that state five different ways, and on Microsoft Foundry the docs and the API use the word to mean opposite things.Research notes13 min read

A loose hand-inked clock dial on steel ledger paper with three hands from one pivot, two in ink and the third reaching past the rim in cobalt

Aug 31, 2026The same AI model retires on three different datesClaude 3 Haiku has three retirement dates across three routes, 143 days apart. Opus 4.1 has two and a pricing tier in between. The receipts, the policy language that causes it, and the ten-model table.Research notes15 min read

Torn black notepad leaf on rust ledger paper, two cobalt blots at the torn corner

Aug 27, 2026OpenAI Codex deprecation: the date, the reason, and what replaced itOpenAI shut down the original Codex API in March 2023 with three days' notice. The dates, the stated reason, what replaced it, and why the name now means something else.Research notes13 min read

Hand-inked Cerebras concentric C on blueprint ledger paper, cobalt stain in the open mouth

Aug 18, 2026Fast on a wafer is not a shortlistCerebras is quicker than Groq on the models both host. The decision is which weights sit on the wafer, and whether your agent is actually waiting on tokens.Pricing and cost11 min read

Hand-inked Groq lightning bolt on bone ledger paper, cobalt blot at the tip

Aug 18, 2026Groq is not a modelAfter Nvidia licensed the chip and hired the team, what is left is a fast, narrow catalog. Most people searching “groq” still think they are buying a brain.Pricing and cost11 min read

Hand-inked Kimi K on manila ledger paper, the raised dot a cobalt blot

Aug 18, 2026K3 is open. You still cannot run it.Moonshot’s 2.8T flagship sits next to the closed frontier. Open here means downloadable shards and a license, not a machine you own.Benchmark explainers12 min read

Two hand-inked Claude asterisks on sage ledger paper: a thin outline beside a heavier mark with a cobalt center

Aug 18, 2026Opus 5 ate the default. Fable is waiting.Claude Opus 5 is Fable-adjacent at $5/$25. That is the default now. Fable 5 still has a job, and 5.1 has to reopen a gap or admit the specialty is gone.Model comparisons13 min read

A short ink dash beside seven thin ticks on steel ledger paper, cobalt blot on the dash

Aug 13, 2026We measure the session. They sell the week.Grok Bot, Hermes, and OpenClaw keep work going across sessions. Their hosting and controls differ; a session benchmark does not measure week-long reliability.Benchmark explainers10 min read

One ledger row circled in black ink on linen paper, cobalt blot on the change

Aug 11, 2026How to Monitor OpenAI API ChangesOpenAI now offers an official release-notes RSS feed, but API changes still span changelog, deprecation, pricing, status, and model-ID surfaces. Here is how to monitor the gaps.Research notes8 min read

Two ink rectangles on cream ledger paper, a small empty one beside a larger one filled with cobalt

Aug 6, 2026Claude Pro vs Max: Which Plan Is Worth It?At USD web list prices, Claude Pro costs $20 monthly or $200 paid annually. Max costs $100 or $200 monthly. Use interruptions and Fable credit spend to decide.Pricing and cost7 min read

A single hand-inked Claude asterisk on slate ledger paper, cobalt at the center

Aug 4, 2026Claude Opus 5 changes the Claude defaultClaude Opus 5 hits 96% on SWE-bench Verified at Opus 4.8's $5/$25 price. Fable 5 still wins harder repo tests, so the upgrade case is not universal.Benchmark explainers8 min read

A simple open mouth in black brush ink on rose ledger paper, cobalt stain in the opening

Aug 3, 2026AI Voice Agents for Customer ServiceCustomer-service voice agents work best on bounded, tool-backed workflows. This guide covers architecture, model choice, rollout gates, and a transparent cost example.Pricing and cost11 min read

Hand-inked magnifying glass on ochre ledger paper, cobalt blot in the lens

Aug 3, 2026Best Enterprise AI Search ToolsGlean leads for a managed company-wide rollout, but Google, Perplexity, Microsoft, Elastic, Onyx, Coveo, Guru, and SearchBlox fit different search estates.Model comparisons13 min read

Three ranking discs on periwinkle ledger paper, the last filled with a cobalt blot

Aug 3, 2026Diffusion LLMs Ranked by EvidenceMercury 2 and Celeris-1 have comparable hosted runtime rows. Gemini Diffusion, DiffusionGemma, LLaDA, and Dream remain provider or research evidence.Model comparisons9 min read

A straight ink stroke beside a circular scribble on terracotta ledger paper

Aug 3, 2026Diffusion vs Autoregressive Language ModelsAutoregressive LLMs generate left to right. Diffusion LLMs revise multiple token positions over denoising steps. Architecture changes the speed trade, not the need to test.Research notes10 min read

Eight ink hashes on teal ledger paper, a magnifying glass over one with a cobalt blot

Aug 3, 2026Nine Glean Alternatives for Enterprise SearchThe right Glean alternative depends on the estate: Microsoft, Google Cloud, Elastic, customer service, verified knowledge, web research, open source, or private deployment.Research notes13 min read

A magnifying glass on rust ledger paper, a retrieve bracket and dash inside the lens

Aug 3, 2026How LLM Enterprise Search WorksLLM enterprise search retrieves permission-safe evidence before generating an answer. Connectors, identity, ranking, citations, and abstention matter more than the chat box.Research notes10 min read

A small ink dash beside a hand-inked Gemini star on seafoam ledger paper

Aug 3, 2026Mercury 2 vs Gemini LiveMercury 2 is a cheap text reasoning layer for a cascaded voice stack. Gemini 3.1 Flash Live Preview is native audio with preview-model risk.Model comparisons11 min read

A small Mercury mark beside a hand-inked OpenAI blossom on brick ledger paper

Aug 3, 2026Mercury 2 vs GPT-Realtime 2.1Mercury 2 is the cheaper text model for a cascaded voice stack. GPT-Realtime 2.1 is the native speech-to-speech choice. The latency numbers are not interchangeable.Model comparisons11 min read

Hand-inked clock on fog ledger paper with a single hand and a cobalt blot at the tip

Aug 3, 2026Time to First Token ExplainedTTFT measures the wait before a model begins answering. It is not output speed or full application latency, and it should be measured as a distribution.Research notes11 min read

Expanding sound-wave arcs on ivory ledger paper, cobalt blot at the origin

Aug 3, 2026What Is an AI Voice Agent?An AI voice agent listens, decides, uses tools, and speaks inside a live conversation. Its quality comes from the full loop, not one model.Research notes10 min read

Hand-inked eye on olive ledger paper with a cobalt iris

Jul 28, 2026Best LLM Observability ToolsCompare six LLM observability tools by documented workflow, privacy, and billing units. A trace-to-test example separates valid JSON from a correct answer.Engineering guides17 min read

Eleven ink ticks and one X on putty ledger paper, cobalt blot on the regression

Jul 26, 2026Twelve Prompt Fixes, Including One RegressionWe ran twelve before-and-after AI prompt examples three times each. Revised prompts improved bounded work overall, but one detailed rewrite made the result worse.Research notes9 min read

Jul 26, 2026Celeris-1 Diffusion LLM for Faster Agentic AI WorkflowsCeleris-1 is a diffusion LLM built for low-latency agentic AI workflows. Review its provider-run speed tests, structured tasks, and OpenAI-compatible API.Sponsored research · Celeris9 min read

A draft scribble kept inside a black ink box on taupe ledger paper, cobalt inside the box

Jul 26, 2026Marketing Drafts That Stay Inside the FactsWe tested 24 marketing drafts against one product source ledger. The useful outcome was copy that followed the facts and requested format before editorial review.Research notes8 min read

An empty ink frame wrapped by a brush stroke on stone ledger paper, cobalt on the outer stroke

Jul 26, 2026Write Prompts Around the Finished WorkA practical six-part method for writing AI prompts that produce testable work: outcome, context, sources, constraints, output contract, and acceptance check.Research notes9 min read

Ink tally groups above a circled cell on ash ledger paper, cobalt in the test-set circle

Jul 26, 2026Four Prompt Optimizers, One Test SetWe attempted 48 public prompt-optimizer rewrites and ran every usable artifact three times. Shared cases tied on accepted outputs; availability was the observed difference.Research notes8 min read

A heavy brush-ink check mark on bone ledger paper, cobalt at the base

Jul 26, 2026Coding Changes That Survive ReviewWe ran 24 AI coding changes against a frozen repository. Better prompts did not improve the test-pass rate; they reduced unrelated code and review surface.Research notes9 min read

Hand-inked hourglass on dusk ledger paper, cobalt pooled in the bottom

Jul 24, 2026The Latency Tax in Agentic WorkflowsAgentic workflows compound latency across model calls, tools, retries, and handoffs. Learn the core patterns, see practical examples, trace the critical path, and build a faster agentic AI workflow without sacrificing reliability.Research notes18 min read

Hand-inked pencil on blush ledger paper, cobalt stain at the tip

Jul 20, 2026How to Optimize an AI PromptOptimize AI prompts with a test loop: preserve facts and constraints, define the output contract, test representative inputs, and fix one failure at a time.Research notes10 min read

A large brush-ink plus sign on moss ledger paper, cobalt blot at the crossing

Jul 17, 2026Is ChatGPT Plus Worth It?As of July 17, 2026, the $8 Go tier carries the same 160-messages-per-3-hours chat allowance as Plus, and OpenAI's own rate card budgets $100–$200 a month for a developer using Codex. We read every current OpenAI pricing page. Plus is worth $20 for exactly two kinds of user.Pricing and cost9 min read

An unfinished Kimi K on sand ledger paper, cobalt apostrophe, lower arm incomplete

Jul 16, 2026Kimi K3: The Open Model Closing the GapMoonshot released Kimi K3's full 2.8T-parameter weights on July 27, 2026. The open release is real; serving it still demands infrastructure on the scale of the model.Benchmark explainers5 min read

A first ink blot on clay ledger paper, cobalt bleeding around it

Jul 15, 2026Thinking Machines Chose Open Weights FirstWhy Thinking Machines made its first foundation-model release open weight, what Inkling changes for the lab, and where it falls short of the closed frontier.Benchmark explainers12 min read

Jul 14, 2026AI Coding Agents Need ReceiptsOn the July 14, 2026 SERP for 'ai coding agents', Vellum's list ranks Vellum first and Augment Code's list ranks Augment Code first. The one cross-agent table anyone publishes put Codex with GPT-5.6 Sol at 80 the same day. Here is the test we are running before we publish a ranking of our own.Benchmark explainers8 min read

Four hand-inked rubber stamp faces with different cuts in a shallow rack, resting over a large cobalt stain on used putty ledger paper

Jul 11, 2026What Legal AI Benchmarks Actually MeasureFour legal AI tests answer different questions. Compare current Vals scores, research costs, and all-pass grading, with each source's own update date.Benchmark explainers8 min read

Jul 8, 2026What Grok 4.5 Means for the FutureGrok 4.5 is not mainly a leaderboard story. At $2 input and $6 output per million tokens, it is a signal that closed frontier models are getting cheaper without becoming open source.Model comparisons10 min read

Jul 7, 2026ResearchClawBench: Why AI Science Agents Still Miss the PaperBenchLM added ResearchClawBench as a display-only benchmark. Claude Code leads at 21.5 RADS, far below the 50-point paper-reproduction line.Benchmark explainers15 min read

Jul 4, 2026HLE's 50% Ceiling Fell in 2026: What Broke It and What Is LeftFor two years, no AI model crossed 50% on Humanity's Last Exam. As of July 2026, fifteen models have, and the top tool-assisted score is 64.7. What broke the ceiling matters as much as the number.Model comparisons13 min read

Jul 4, 2026The Missing-Benchmark Problem: How Leaderboards Reward Hiding Weak ScoresA reader caught BenchLM ranking Qwen3.7 Max below its own cheaper sibling. The bug was not a data error. It was the averaging method almost every LLM leaderboard uses, and fixing it moved 170 scores.Benchmark explainers13 min read

Jul 4, 2026What AI Labs Don't Publish: The Benchmark Disclosure GapA July audit found far more published coding evidence than multilingual evidence among leading models. The disclosure gap still shapes rankings under BenchAlign.Benchmark explainers13 min read

Jul 2, 2026Introducing BenchLM Stats: Citable LLM Data, Regenerated on Every UpdateWe packaged the data behind BenchLM into 26 citable statistics across six pages — model prices, release cadence, context windows, benchmark saturation, open-source share, and market share. Every number is a self-contained, dated sentence generated from the live dataset, with a stable anchor URL you can cite.Research notes5 min read

Jul 2, 2026Best AI Web Scraping ToolsAs of July 28, 2026, Browse AI is our no-code shortlist, Firecrawl fits site-to-markdown jobs, and Apify offers the broadest developer platform of the three. We map documented fit and state plainly that we have not run a head-to-head extraction benchmark.Engineering guides9 min read

Jul 2, 2026Best Hosting Platforms for AI Apps in 2026: The Deploy LayerAs of July 2026, Netlify is our pick for short-running prototypes, Cloudflare for edge-scale production, and Railway or Fly.io for long-running backends. The limits that decide the choice.Engineering guides8 min read

Jul 2, 2026Best Text-to-Speech APIs for AI AppsAs of October 13, 2026, listeners favor Cartesia Sonic 3.6 among the TTS APIs we priced. Delay, billing units, and commercial rights decide the rest.Engineering guides11 min read

Jul 2, 2026How to Deploy an AI App in 2026: From Model Pick to Production URLA practical guide to deploying AI apps and LLM-powered products — the model layer vs. the app layer, two runtime constraints, three operational checks, and the production pattern we use.Engineering guides15 min read

Jun 27, 2026Fable 5 vs GPT-5.6: Two Bets on Where the Frontier Goes NextAnthropic shipped Fable 5 and Mythos 5 at $10/$50. OpenAI moved GPT-5.6 Sol, Terra, and Luna to general availability on July 9 at a lower price ladder. The launch gap is the story of the 2026 AI market.Model comparisons16 min read

Jun 19, 2026How We Keep Pricing Data CurrentThe current pipeline starts with official first-party sources, a curated registry, validation, and human review. Downstream publication is automated; provider-page scraping is not.Engineering guides11 min read

Jun 12, 2026Best LLMs for Voice AI Agents (2026)Telnyx has the lowest median phone delay, ElevenLabs the lowest p95, in the July 31 run. Pick the stack from your delay budget, then the model from your calls.Model comparisons12 min read

Jun 12, 2026How to Get Cited by ChatGPT, Perplexity & Claude in 2026: AEO from the TrenchesA practitioner's guide to getting cited by ChatGPT, Perplexity, and Claude — the exact AEO/GEO changes we shipped on BenchLM: quotable lines, Dataset schema, llms.txt, AI-crawler access, and the tooling we use to find what to answer.Research notes12 min read

Jun 9, 2026Claude Fable 5 and Mythos 5: The Future of AI Is Gated IntelligenceAnthropic's Claude Fable 5 brings Mythos-class capability to public users, while Claude Mythos 5 remains trusted-access. The benchmark story is strong, but the real shift is capability-gated deployment.Benchmark explainers16 min read

$Best LLM for Math 2026: FrontierMath, AIME & HMMT Rankings article cover$ May 12, 2026Best LLM for Math 2026: FrontierMath, AIME & HMMT RankingsBest LLM for math 2026: live FrontierMath, AIME 2026 and HMMT February 2026 tables, why AIME 2025 and MATH-500 no longer pick a winner, and picks by use case.Model comparisons10 min read

May 12, 2026Perceptron Mk1 and Frontier Video Models: The Complete Guide to Video Understanding AIA complete guide to Perceptron Mk1, frontier video understanding models, video AI benchmarks, and where video-language models are headed next.Benchmark explainers52 min read

May 5, 2026ProgramBench Benchmark Explained: Can LLMs Rebuild Programs From Binaries?ProgramBench is a new LLM coding benchmark where agents rebuild full programs from a compiled binary and documentation. See scores, how it differs from SWE-bench, and why all public models are 0% resolved.Benchmark explainers17 min read

Apr 27, 2026ARC-AGI-2 Explained: The Hardest Public Reasoning BenchmarkARC-AGI-2 measures fluid intelligence through visual grid puzzles that can't be solved by memorization. Here's how it works, what scores mean, and where current frontier models stand.Benchmark explainers17 min read

Apr 24, 2026Advertised vs Effective Context Windows: What 1M-Token Claims HideFour frontier LLMs advertise 1M+ tokens, but advertised and effective context are different numbers. DeepSeek V4 Pro's 384K output changes generation workflows; Gemini leads effective-context evals. What the headline claims hide.Model comparisons14 min read

Apr 24, 2026DeepSeek V4 Pro vs Claude Opus 4.7 vs GPT-5.5: The Frontier in April 2026Three frontier flagships launched in eight days. Here is how DeepSeek V4 Pro, GPT-5.5, and Claude Opus 4.7 compare on benchmarks, current cost, and real use.Model comparisons17 min read

Apr 9, 2026GPT-5 vs Gemini in 2026: Full Benchmark BreakdownGPT-5.6 Sol vs Gemini 3.5 Flash on current BenchAlign overall, coding, agentic, price, context, and evidence data.Model comparisons8 min read

Apr 7, 2026Mythos Preview is the first frontier model Anthropic decided not to ship. The benchmarks show why.Claude Mythos Preview beats Opus 4.6 by double digits on every coding benchmark Anthropic released. Then they shelved it. Here's what the numbers actually show, and why the shipping decision matters more than the launch.Benchmark explainers12 min read

Apr 6, 2026Choosing the Best LLM for RAGWe do not publish a synthetic RAG ranking. The best model is the cheapest candidate that clears your grounded-answer eval; live long-context and instruction-following rows build the shortlist.Model comparisons11 min read

Apr 6, 2026The Best Writing Model Saves the EditThe best LLM for writing is a task-based shortlist. Start with three current models, then compare factual failures, constraint failures, and editing time.Model comparisons9 min read

Apr 4, 2026How to Choose an LLM in 2026: Best by Use CaseChoose the right LLM using current overall, coding, agentic, open-weight, price, access, and evidence data instead of one universal winner.Model comparisons9 min read

Mar 30, 2026Why Chinese LLM Rankings DisagreeThe provisional lane picked Qwen3.7 Max; BenchAlign v5 picked MiMo-V2.5-Pro. A dated audit of the contract, identity, and creator-filter split.Model comparisons7 min read

Mar 30, 2026ChatGPT vs Claude vs Gemini: Which to Pick for Each TaskClaude leads our October 2026 evidence on coding, research, and presentations, ChatGPT leads on reasoning, and blind voters favor Gemini for chat. Plus what each plan gets you.Model comparisons8 min read

Mar 26, 2026How LLM Token Pricing Works: A Complete Guide to API Costs in 2026Learn how LLM API pricing works — from tokens, input/output costs, and reasoning tokens to vision, embedding, and fine-tuning pricing. Includes real cost examples, free tiers, and 6 strategies to cut your AI spend.Pricing and cost15 min read

Mar 24, 2026React Native Evals: The Mobile App Coding Benchmark ExplainedReact Native Evals measures whether AI coding models can complete real React Native implementation tasks across navigation, animation, and async state. Here's what it tests, why it matters, and how it differs from SWE-bench and LiveCodeBench.Benchmark explainers7 min read

Mar 22, 2026State of LLM Benchmarks (July 2026): 296 Evals TrackedThe July 2026 state of LLM benchmarks: 296 tracked evaluations, current overall, coding, agentic, and open-weight leaders, plus what BenchAlign changes about missing data.Model comparisons10 min read

Mar 18, 2026Are AI Benchmarks Reliable? The Data Contamination ProblemAI benchmarks are useful but flawed. Data contamination inflates scores when models train on test questions. Here's how it works, which benchmarks resist it, and how BenchLM accounts for reliability.Benchmark explainers9 min read

Mar 18, 2026Best Budget LLMs in 2026: Every Model Under $1.50 per 1M Tokens, RankedWhich budget LLM should you use in 2026? A live ranking of every model at or under $1.50 per million input tokens by BenchAlign score, evidence and price, plus picks by workload.Model comparisons12 min read

Mar 12, 2026BrowseComp Explained: How We Measure Web Research AgentsBrowseComp evaluates whether AI models can search the web, gather evidence, and answer research questions instead of relying only on latent knowledge.Benchmark explainers6 min read

Mar 12, 2026Claude Opus 4.6 vs GPT-5.4: Full Benchmark Breakdown (2026)Claude Opus 4.6 vs GPT-5.4 with current BenchAlign scores, coding and agentic evidence, raw benchmarks, pricing, and the newer models you should also consider.Model comparisons8 min read

Mar 12, 2026LLM Pricing 2026: Every Model, $0.11–$50 per 1M TokensHow to actually compare LLM API pricing — what blended cost means, which discounts matter (caching, batch), and how to find the cheapest model for your workload. With current rates for GPT-5, Claude, Gemini, DeepSeek, and more.Model comparisons8 min read

Mar 12, 2026OSWorld-Verified Explained: How We Measure Computer-Use ModelsHow OSWorld-Verified tests computer-use systems, how its sourced results should be compared, and why OSWorld 2.0 is a separate protocol.Benchmark explainers7 min read

Mar 12, 2026Terminal-Bench 2.0 Explained: How We Measure Agentic CodingTerminal-Bench 2.0 measures whether AI models can work through real terminal-based coding and ops workflows instead of just answering in chat.Benchmark explainers6 min read

Mar 12, 2026What Do LLM Benchmarks Actually Measure?LLM benchmarks don't measure intelligence. They measure specific, narrow abilities under controlled conditions. Here's what each benchmark type actually tests — and what it misses.Benchmark explainers10 min read

Mar 7, 2026AIME & HMMT: Can AI Models Do Competition Math?AIME and HMMT are high school math olympiad competitions now used to benchmark AI. Frontier models score 95-99% — competition math is effectively solved. Here's what that means.Benchmark explainers10 min read

Mar 7, 2026Arena Elo Explained: How LMArena Chatbot Rankings WorkArena (formerly Chatbot Arena / LMArena) ranks AI models with Elo ratings from blind human preference votes. How the system works, what scores mean, and how Elo compares to benchmarks.Benchmark explainers10 min read

Mar 7, 2026Claude Opus 4.6 vs GPT-5.4: Where Each Model WinsClaude Opus 4.6 vs GPT-5.4 on current overall, coding, agentic, raw benchmark, and pricing data, with the evidence caveats that decide the close calls.Model comparisons6 min read

Mar 7, 2026GPQA Diamond: The PhD-Level Science BenchmarkGPQA tests AI models with graduate-level questions in biology, physics, and chemistry that are 'Google-proof' — even skilled non-experts with internet access can't answer them. Here's how it works.Benchmark explainers10 min read

Mar 7, 2026HLE (Humanity's Last Exam): The Hardest BenchmarkHumanity's Last Exam is crowdsourced from thousands of domain experts and designed to probe the absolute frontier of AI. Top models still top out under 65%. Here's why HLE matters.Benchmark explainers10 min read

Mar 7, 2026LiveCodeBench: Why Static Coding Benchmarks Aren't EnoughHow LiveCodeBench uses fresh contest problems, why release and metric labels matter, and what its scores do not say about repository engineering.Benchmark explainers8 min read

Mar 7, 2026MMLU vs MMLU-Pro: What Changed and Why It MattersMMLU and MMLU-Pro are the most cited knowledge benchmarks in AI. Here's what each measures, why MMLU is saturated, and why MMLU-Pro is the better discriminator in 2026.Benchmark explainers7 min read

Mar 7, 2026SWE-bench Explained: How We Measure Real-World CodingSWE-bench Verified tests AI models on resolving real GitHub issues from Django, Flask, and scikit-learn. Here's how it works, why it matters, and which models score highest.Benchmark explainers7 min read

Mar 7, 2026What Is HumanEval? The Coding Benchmark ExplainedHumanEval tests whether AI models can generate correct Python functions from docstrings. Here's what it measures, why it's nearly saturated, and which benchmarks matter more in 2026.Benchmark explainers6 min read

An uneven ink caliper on an ivory ledger sheet, with a cobalt stain spreading from one jaw.

Aug 22, 2025Test the replacement on what already failsA replacement model can raise the benchmark score and still break the field your application needs. Build a small workload evaluation from your own failing cases, run both configurations against the same checks, and keep the incumbent when the numbers say so. With a downloadable kit and two filed runs.Benchmark explainers12 min read

The Complete Guide to LLM Benchmarking: Everything You Need to Know article cover

Aug 22, 2025The Complete Guide to LLM Benchmarking: Everything You Need to KnowEverything you need to know about LLM benchmarking — what benchmarks measure, how to choose the right ones, common pitfalls, and how to interpret results for real-world model selection.Benchmark explainers15 min read

How to Interpret LLM Benchmark Results: A Practical Guide article cover

Aug 22, 2025How to Interpret LLM Benchmark Results: A Practical GuideHow to read LLM benchmark scores correctly — what differences are meaningful, what to ignore, common misinterpretations, and how to translate benchmark data into model selection decisions.Benchmark explainers10 min read

Get the BenchLM weekly brief

New analysis, material benchmark and price changes, and one model decision worth revisiting.

Read a sample issue

Join 6,100+ subscribers.