AI Coding Agents Need Receipts

Ask Google which AI coding agent to buy and the top of the page answers with lists written by the contestants. On the July 14, 2026 US results for "ai coding agents," Vellum's ranking puts Vellum first, Augment Code's ranking puts Augment Code first, and the AI Overview stitches its summary from both. The one genuinely cross-agent table in public, Artificial Analysis's Coding Agent Index, had Codex running GPT-5.6 Sol on top at 80 when we read it the same day; we mirror its June snapshot as display-only evidence. That number is real. It is also not enough to spend money on, and this post is about what would be.

The lists rank their authors

We pulled the live results page before writing this, and the pattern took no digging. Position 1 organic belongs to an engineering-analytics vendor's "real-world developer reviews." The AI Overview cites six lists, two of which award first place to their own publisher. A third ranks agents by Terminal-Bench scores its author never ran, and quotes a leader a full model generation has passed since. Position 2 belongs to a Reddit thread.

The Reddit placement is the interesting one. Google ranking sixty developer comments above most of the professional lists is a verdict on those lists: honest anecdotes currently beat interested rankings, because at least the anecdotes were produced by someone running the tools on their own work. The market is starving for evidence, not opinions.

None of this makes the vendor lists useless. It makes them testimony from interested parties, which is a kind of evidence you read differently, the way a judge reads a defendant's account of the crash. What the SERP is missing is the other kind: the same tasks, on the same repository, across every agent, with the failures published.

What the one real cross-agent table shows

Artificial Analysis runs the closest thing to that today. Its Coding Agent Index scores the agent product (harness, host model, and execution settings together) across DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnA, and publishes cost, time, and token metadata beside each score. Our mirror holds the June 2026 snapshot, 34 rows. The best row per agent in that snapshot:

Table 1 Download CSV

Agent	Host model (best row)	Index	Avg time/task	Avg cost/task
Codex	GPT-5.6 Sol (xhigh)	78.7	7.4 min	not reported
Claude Code	Opus 4.6 (medium)	71.1	8.0 min	$1.26
Cursor CLI	GPT-5.4 (medium)	68.8	8.3 min	$1.52
Opencode	Opus 4.7 (medium)	64.5	12.2 min	$2.93

Now watch what one month does. We re-read the live index while writing this, July 14: Codex with GPT-5.6 Sol at max reasoning leads at 80, Claude Code now pairs with Fable 5 at 77, and Gemini CLI has joined near the bottom at 43. Two of the four rows in our own dated table above are already superseded, OpenAI is quoting the index in its GPT-5.6 launch post, and per-task cost on the live page spans $0.27 to $11.80 depending on the pairing. Any list that transcribed this leaderboard in June is wrong in July. That is not a flaw in the index; it is the reason transcription is not evaluation.

Read the sampling before the podium, too. Twenty of the 34 June rows were Codex variants, so "Codex leads" partly means "Codex is the most-sampled," and most rows rest on two or three evaluation runs, which is a coin flipped twice. The live page's sharpest new evidence cuts the other way entirely: holding the same host model constant (Opus 4.7), the harness ordering is Opencode 65, Cursor CLI 60, Claude Code 57. The harness alone moves the score eight points, and the harness that wins with one model is not the harness winning the headline table with another.

We mirror this index on its own benchmark page and keep it out of our weighted model rankings, because a row that bundles a harness with a model can't be compared against rows that measure models alone. It is the best public evidence in the category and it still cannot tell you what a failed Tuesday afternoon costs.

Six numbers that would actually decide it

Benchmark pass rate is one input to a purchase decision that involves at least six. This is what a ranking would report if it were built by someone who has to live with the agent afterward:

Table 2 Download CSV

Measurement	The question it answers
Pass rate vs hidden acceptance tests	Did it actually fix the thing?
Interventions per run	How often did a human unblock it?
Elapsed wall-clock	Can you run it inside a workday?
Tokens and metered cost	What does a task cost at your plan's prices?
Diff size vs a human reference fix	Did it fix the bug or remodel the module?
Rollback burden after failure	How long to get a clean tree back?

The last row is the one nobody publishes, and it dominates real operating cost. A failed agent run is not free. The cheap failure is a diff you discard with one command; the expensive one edited your CI config, touched four unrelated files, and left a service running — and you find out which kind you got forty minutes later. Two agents with identical pass rates can sit at opposite ends of that bill.

Intervention counting needs discipline too, or it turns into marketing. Approving a permission prompt, answering a direct question, and grabbing the wheel after a wrong turn are different events, and a run that needed the third is not "autonomous" no matter how good the final diff looks.

The fixture we are running

So we wrote the test before the verdict. One real open-source repository, pinned to a single commit. Eight tasks from its actual issue history — three bug fixes with reproducing tests, two small features against written specs, a behavior-pinned refactor, a breaking dependency upgrade, and a test-writing task graded against seeded mutants. Acceptance tests stay hidden from the agent. Every agent gets identical prompt bytes, its out-of-the-box model and settings, and a 45-minute cap. Three runs per task per agent, medians reported, all runs published.

The roster rule: shipped, active products an individual can buy that run against a local repository — Claude Code, Codex CLI, Cursor CLI, GitHub Copilot's cloud agent, Gemini CLI, Opencode. Framework SDKs and cloud-only PR bots wait for a different fixture, and Aider, the category's old default, sits this one out because it hasn't shipped a release since August 2025.

Everything ships with the results: task definitions, scoring rules, one branch per run, versioned agent configurations, and the failed runs at the same level of detail as the passes. If an agent destroys the working tree, that run gets published with the cleanup timed.

We are not publishing a ranking until those runs exist.

That sentence is the whole editorial position. A table of six agents with scores we made up this afternoon would rank better than this page does, for a while. It would also be one more list the next version of this post has to apologize for.

How to read any agent list until then

Four checks, thirty seconds:

1. Who ranks first, and who published the list? If the answers match, you are reading an ad with headers.
2. Did the author run anything? Look for task definitions and failure cases. "We tested on real codebases" without either is a vibe with a byline.
3. Whose benchmark numbers are these? Reused public scores date fast (the SERP's current Terminal-Bench quote still names GPT-5.5) and say nothing about interventions or cleanup.
4. Are failures reported? An evaluation with no failed runs is a brochure. Every agent fails; the honest question is how expensively.

Our Claude Code vs Cursor and Cursor vs Copilot pages hold the two-tool decisions, the coding leaderboard holds model-level scores, and the Terminal-Bench 2.0 explainer covers the benchmark the listicles keep borrowing. The receipts for this category are coming from the fixture above.

The next update to this page will be rows, not adjectives.