How the voice benchmark pages are built

Launch page with 6 featured and 15 additional recordings, October 2026. Block chose and recorded these conversations, and it reports no quantitative evaluation. Faster-than-real-time inference, local MacBook Pro M4 Max inference, and single GB200 NVL72 training are Block claims that have not been independently checked. Nothing here enters weighted text-model scores.

Provider-published selected figures; no shared overall score. These are selected figures published by ElevenLabs and attributed to an independent evaluator, rather than a complete independently retrieved leaderboard. Big Bench Audio, Full-Duplex-Bench, Speech Agent Arena Elo, task success, and latency have different units and protocols. The page gives no sample counts, uncertainty, evaluation dates, task-success rubric, or benchmark version for Full-Duplex-Bench. Time to first audio is the average measured on Big Bench Audio. Do not merge these figures with similarly named existing benchmark versions or rank Mini against Max on metrics published for only one of them. The third-party figures are not licensed here for Data API redistribution.

Exact values in the chart data of the October 20, 2026 launch post. All results are Mistral-run. FLEURS macro-averages use CER for Japanese, Korean, Thai, and Chinese. Diarization averages AMI-IHM, AMI-SDM, Switchboard, Switchboard CallHome, and AISHELL-4, with 30-minute AMI and AISHELL versions. Realtime rows trade latency for accuracy; the Mini Transcribe 3 row on FLEURS 35 is an offline baseline. Voice Code Bench is a BeSimple AI test of numbers and names in support calls; the multilingual and expressive scores are judged by gemini-3.8-flash. The launch post and docs disagree on TTS language count (35+ vs 9) and realtime minimum latency (240 ms charted, 80 ms documented). Competitor rows from Mistral charts are not carried here. These results do not enter text-model or voice-agent rankings.

Exact values in the Hugging Face model cards; the audio and ASR cards publish them as table images. All results are inclusionAI-run or developer-reported by inclusionAI. Text-to-speech figures use a unified ASR-based WER or CER protocol over each open model at its recommended settings; speaker similarity (SIM) trails content accuracy, and Korean CER for Ming-3.0-tts-tiny (6.000%) is the clearest multilingual gap. The audio and ASR figures are read from images on the model cards; results marked with an asterisk there are self-evaluated. Competitor rows are not carried here. These results do not enter text-model or voice-agent rankings.

Exact values in the Hugging Face model cards and the launch post. Liquid AI measured every result, including the competitor columns not carried here, with each model’s own chat template and greedy text decoding. English ASR averages the ESB / Open ASR suite; multilingual ASR averages FLEURS over nine languages, with CER for Chinese, Japanese and Korean. VoiceBench is the mean of 9 subsets with the three 1-5 judge scores multiplied by 20. The tool-calling score is an internal evaluation without a public task set. These results do not enter text-model or voice-agent rankings.

Exact values in the Hugging Face model card tables. NVIDIA ran every result itself. Transcription, translation and language identification use reasoning off at temperature 0; audio understanding and text reasoning use reasoning on at temperature 0.6, top-p 0.95, with a 16,384-token reasoning budget. The English Open ASR mean is NVIDIA’s equal-weight mean of nine sets, including Monsoon English and URGENT24, so compare it only with runs over the same set list. The multilingual board uses WER throughout, while standalone FLEURS uses CER for Chinese, Japanese, Korean, Thai and Vietnamese, so those numbers are not directly comparable. The card’s per-language FLEURS, CoVoST2, language-identification and translation tables are summarised, not copied in full. The “next best” comparisons in NVIDIA’s summary tables are not carried. These results do not enter text-model or voice-agent rankings.

Exact provider tables in the model card and October 2026 launch blog. IFM evaluated Jais-ASR using the public leaderboard script. These are provider-reported measurements, not independently verified leaderboard submissions. The six-set Arabic macro-average, Mixat, and internal Arabic Speech Benchmark have different samples and aggregation rules and are not pooled. Speech error rates stay separate from weighted text-model scores. October 15 is the source retrieval date, not an established evaluation or first-release day.

Provider tables published October 14, 2026. HeyGen ran the digital-twin and platform studies. The 43-person ratings and 6,153-vote preference shares describe different studies and model modes. These observations do not enter weighted text-model scores. The research publication date does not establish first availability.

Exact accuracy stated in the October 14 launch post. The 74% figure is a provider-reported EVA-Bench claim, not an independently refreshed owner result here. Voice results stay separate from weighted text-model scores.

Specifications published October 1, 2026; no independent listening-test score is included. The model page reports about 550 ms inference for Voice-2.1 and 45 ms for Flash. The launch gives a separate Flash example: 45 seconds of audio generated with 150 ms end-to-end latency. These timings measure different stages. Microsoft reports 23 languages; its launch says 26 locales while the model page lists 28 labels, so the locale total is not reconciled here. These specifications do not establish an independent speech-quality rank.

Provider specifications published October 1, 2026; evaluator measurements are linked separately. Microsoft reports first hypotheses just over 100 ms after receiving audio. That timer differs from transcript latency measured after detected speech end. The non-streaming model's FLEURS result does not establish this streaming release's accuracy.

Exact evaluator payload retained in private evidence; percentages rounded to one decimal and seconds to two decimals. This single configuration capture does not mirror the full leaderboard. The index weights AA-AgentTalk at 50%, VoxPopuli at 25%, and Earnings22 at 25%. Latency starts at SileroVAD-detected speech end; first partial is the first transcript-bearing event afterward. Missing endpoint finals can use the documented partial fallback. Evaluation date, hardware, and confidence intervals are not published in this row. The capture date does not establish when the evaluation ran.

Exact tables in the launch report, retrieved September 30, 2026; its accuracy table is also on the official model card. Fermion ran Phonon-2, Phonon-1, and Parakeet Redux with the Open ASR Leaderboard code on full test sets. The other accuracy rows are leaderboard figures quoted by Fermion, without independent re-verification here. Some model cards report different runs; those values are not mixed into this comparison. Voxtral's download size is an estimate at 16 bits. Throughput includes log-mel processing and excludes model load; batch-128 GPU results are separate from single-stream results. The Mac runtime comparison uses 20 dictations totaling 797 seconds at each runtime's defaults. Download size does not measure runtime memory. These results do not enter text-model or voice-agent rankings.

Google states the Flash TTS design and accent scores and the two quality-index placements in prose; it does not state Flash-Lite’s underlying quality score.. These are Google-reported placements on third-party Hume evaluations, not independently verified results in this catalog. Flash-Lite has no published exact quality-index value in the launch prose. Voice evaluations remain separate from weighted text-model rankings.

The short-phrase pair and the pricing are stated in the post’s prose. The four internal charts and the multilingual comparison render client-side, so their per-category values are not published in a readable form.. The internal sets are xAI’s own production traffic, not a released dataset, so nothing here is independently reproducible. The four per-category charts and the multilingual bar chart are drawn in the browser and carry no readable figures, so BenchLM stores only the values the post states in prose and does not read numbers off the plots. The leaderboard rank is an xAI claim about a third-party board that BenchLM does not track as a source; it is not a BenchLM measurement. These transcription results stay separate from BenchLM’s weighted text-model ranking.

Exact figures tabulated in the official launch post for all five compared systems. Qwen ran every system in this table itself and chose the harness for each agent row: WildClawBench-MM and AgenticVBench use Claude Code, UniClawBench uses OpenClaw, OmniGAIA uses no harness, and the static-versus-agent comparison uses Qwen Code. WildClawBench-MM covers only the multimodal subset of WildClawBench. Competitor runs use provider-specific media settings (Gemini 3.8 Flash at media_resolution=high, Seed 2.0 Lite at max_frame_tokens=384), so the comparison is not independently reproducible. These voice-protocol results stay separate from BenchLM’s weighted text-model ranking.

Exact figures labelled on the charts in the official launch post; the two dictation sets are private to Wispr. Wispr ran every system in this comparison itself, and the two headline sets are its own unreleased dictation data, so the results are not independently reproducible. Competitor runs used no contextual prompting, which is not how several of them are deployed, and the challenge-set winner (Gemini 3.1 Pro) is a frontier multimodal model Wispr describes as unsuitable for real-time dictation.

Exact bar-chart labels in Google’s model evaluation PDF, plus figures stated in the prose of the launch announcement; EVA-Bench scatterplot points are not tabulated. Google’s PDF combines evaluations run by separate teams using different APIs and effort settings. The Extended Thinking task rows use high effort, and the EVA-Bench scatterplot labels are plotted without exact figures. These voice-protocol results do not enter weighted text-model scores, and the plotted audio-hour costs are not canonical API prices.

Specifications only; OpenAI publishes no evaluation numbers for this model. This entry records what OpenAI documents for gpt-audio-1.5 rather than a benchmark result. BenchLM assigns no score.

Specifications only. This entry records what OpenAI documents for gpt-transcribe rather than a benchmark result. BenchLM assigns no score.

Specifications only. This entry records what OpenAI documents for gpt-live-transcribe rather than a benchmark result. BenchLM assigns no score.

Exact figures published in the official launch post. These are provider-reported launch results against Google’s own prior model; the post gives no per-language breakdown or competitor rows.

Exact figures published in the official launch post. These are provider-reported launch results; the post gives competitor rankings and speed ratios without their exact WER values.

Exact figures published in the official Foundry post. Provider-reported results against Microsoft’s own previous model.

Relative claims only. Provider claims relative to Microsoft’s own MAI-Voice-2; no listening test or leaderboard result is published.

Exact figures published in the official launch post. Provider-reported results; the Elo figures are Cartesia’s citation of an independent speech leaderboard at launch and the preference tests are Cartesia-run.

Rank and latency published in the official launch post and docs; no Elo value given. Provider-reported; Inworld cites the Speech Arena rank without the underlying Elo.

Documented figures only. Provider-documented latency; no arena or listening-test result is published for the Flash variant.

Relative claim only. Provider claim without a published benchmark table or competitor values.

Tencent’s site is descriptive; the WER values come from launch coverage and are marked secondary. The WER figures are reported by launch coverage (BigGo, August 5, 2026) rather than a Tencent page BenchLM could retrieve; treat them as secondary.

Latency figure published; accuracy stated relative to Nova-3 without a WER value. Provider-reported launch figures; no independent WER table is published.

Latency and coverage published; accuracy stated as monolingual-grade without a WER value. Provider-reported launch figures; no WER table is published.

Documented figures only; no listening-test result. Provider-documented performance on reference hardware; no quality benchmark is published.

Exact figures published on the official GGUF model card. Provider-run evaluation on Zyphra’s own set; the blog compares against other systems qualitatively only.

Throughput published in text; WER charts are images. IBM publishes the Open ASR and FFASR leaderboard comparisons only as images, so no WER value is transcribed here.

No evaluation numbers published. Cohere publishes no accuracy figures for this model at the time of tracking.

Evaluation table published only as an image. No numeric results are transcribed because the card publishes them as an image.

Exact figures published on the official model card. Provider-run evaluation on public English test sets.

Benchmark chart published only as an image. No numeric results are transcribed because the card publishes them as an image.

Exact figures embedded in the official launch post charts. These are provider-reported launch results against OpenAI’s own Realtime models, not an independent voice-agent ranking. The Tau3 and Tau Banking rows pair GPT-Live-1 with GPT-6 Astra at medium reasoning effort as the delegated backend, and the Full Duplex Bench v3 rows use the Terra backend at low effort. The Full Duplex Bench versions and runs differ from the v3 paper table BenchLM mirrors separately.

Exact figures published in the official launch post. These are provider-reported launch results, not an independent end-to-end voice-agent ranking. The post does not publish enough per-dataset detail to reconstruct either aggregate from raw runs.

Machine-readable public leaderboard. The leaderboard measures perceived human-likeness for the selected American-English voices and prompt mix. It does not measure factual accuracy, task completion, full-duplex interaction, or multilingual quality.

Machine-readable public leaderboard. Open-ended answers use an automatic model judge. Its overall score combines heterogeneous task scales using the benchmark owner’s method.

Machine-readable benchmark-owner leaderboard. The refreshed table uses the parsed MMAU-v05.15.25 test results. It does not merge older benchmark versions or unverified community-reported rows.

Machine-readable benchmark-owner leaderboard. The overall average spans heterogeneous audio tasks. Inspect speech, voice, spatial, multi-audio, and instruction-following columns before using it to choose a system.

Machine-readable public run records. Scores are judge-derived checks aggregated across recent public runs. Human and synthesized audio remain separate views.

Results transcribed from the official paper. The six-system result table is a fixed paper snapshot, not a continuously updated leaderboard.

Machine-readable benchmark-owner submissions and reproducible evaluation code. Commercial voice APIs and replacement voice IDs are required to reproduce the published setup.

Machine-readable benchmark-owner leaderboard, code, and public dataset. Accuracy and experience are deliberately separate axes; neither should be collapsed into the other.

Machine-readable public API and reproducible raw-call artifacts. This board measures platform-level phone latency, not answer quality or raw speech-model latency. The carrier path and each platform’s deployed stack remain inside the measurement.

Paper and public dataset. Its broad multilingual coverage is primarily synthetic, so it should not stand in for human-audio robustness.

Paper and benchmark resources. This is a diagnostic benchmark for ambiguity and disfluency, not an end-to-end service-agent ranking.