A map of the voice benchmark landscape

xAI’s September 18, 2026 launch post reports a rank on an independent streaming speech-to-text leaderboard, four internal word-error-rate sets drawn from production traffic, and one exact multilingual short-phrase pair against Grok Voice Transcribe 1.0.

Qwen’s September 18, 2026 launch post reports a 32-row audio and audio-visual table for Qwen3.8-Omni-Flash against Qwen3.5-Omni-Plus, Gemini 3.8 Flash, Seed 2.0 Lite, and Muse Spark 1.2, plus a separate static-versus-agent comparison on three video sets.

Wispr’s September 17, 2026 launch post reports word error rates for Canto against five competing transcription systems on two private dictation sets and three public English datasets.

Google’s September 15 model evaluation reports separate results for Gemini 3.8 Live and Gemini 3.8 Live Extended Thinking on spoken agent tasks and a speech-to-speech index. The same-day launch announcement adds a Big Bench Audio score and a Speech Agent Arena standing.

OpenAI’s model page documents the modalities, limits, and pricing of gpt-audio-1.5; it publishes no benchmark scores for the model.

OpenAI’s model page documents endpoints, features, and per-minute pricing for gpt-transcribe; it publishes no word-error-rate table on the page.

OpenAI’s model page documents the realtime-only endpoint, latency controls, and per-minute pricing for gpt-live-transcribe; it publishes no accuracy table on the page.

Google’s August 26, 2026 launch post reports word error rates on an independent transcription leaderboard and FLEURS for pre-recorded and streaming transcription, plus a latency gain over Chirp 3.

Microsoft AI’s September 3, 2026 launch post reports FLEURS word error rate across 60 languages, an independent leaderboard position, and relative speed against competing transcription models.

Microsoft’s June 2, 2026 Foundry post reports the FLEURS word error rate improvement for MAI-Transcribe-1.5 over MAI-Transcribe-1.

Microsoft AI’s July 23, 2026 post positions MAI-Voice-2-Flash as a faster, cheaper sibling of MAI-Voice-2 without publishing benchmark scores.

Cartesia’s August 27, 2026 launch post reports independent controlled-voice Elo ratings, blind preference win rates against Eleven v3, response latency, and locale coverage for Sonic 3.6.

Inworld’s launch post reports an independent speech-arena ranking, time-to-first-audio, and language coverage for Realtime TTS-2.

Inworld’s model documentation reports time-to-first-byte and coverage for the Flash variant of Realtime TTS-2.

ByteDance Seed’s August 5, 2026 announcement describes an end-to-end audio-visual full-duplex model and reports a halving of conversational pacing issues versus cascaded pipelines.

Tencent’s Hunyuan site describes Hy ASR 3.0 preview as a Hy3-based recogniser with context-aware correction and broad dialect coverage; launch coverage cites word error rates near 3%.

Deepgram’s October 2, 2025 launch announcement reports end-of-turn detection latency and Nova-3-level accuracy for Flux.

Deepgram’s April 29, 2026 press release reports end-of-turn latency and language coverage for Flux Multilingual.

Kyutai’s TTS page and repository document Pocket TTS’s size, CPU real-time factor, and first-audio latency.

Zyphra’s ZONOS2 GGUF model card reports word error rate, speaker similarity, and UTMOS for the reference F16 build.

IBM’s model card and Granite 4.2 blog document the model’s size, training data, and throughput; Open ASR leaderboard results are published as images.

Cohere’s catalog and Hugging Face repository document the Arabic transcription model’s lineage and availability; no evaluation table is published.

Microsoft’s model card documents streaming speaker-attributed transcription, hotword support, and ten-language coverage; evaluation results are published as an image.

Qwen’s model card reports word error rates for Qwen3-ASR 1.7B and 0.6B across AMI, Earnings22, GigaSpeech, LibriSpeech, SPGISpeech, and VoxPopuli.

Tencent’s model card documents AuK’s task coverage across speech generation, editing, enhancement, and separation; its benchmark chart is published as an image.

OpenAI’s API launch post reports spoken task success, full-duplex interactivity, tool calling, response quality, and turn-taking latency for GPT-Live-1 against GPT-Realtime-2.1 and GPT-Realtime-2.

Meta’s launch evaluation reports streaming transcription accuracy, speaker diarization, and real-time audio-perception capabilities for Muse Voice Transcribe.

A source-owned index for native audio models, with separate results for spoken reasoning, customer-service task completion, live preference, conversation flow, latency, and audio cost.

One thousand spoken reasoning questions test whether native audio models can answer correctly from audio and how quickly they begin replying.

People hold blind, paired voice conversations and choose the system they prefer; eligible tool-calling conversations also receive a task-success check.

Blind listening comparisons measure how human text-to-speech output sounds across phone-agent, conversational, and explainer speech.

A spoken question-answering suite spanning open-ended, knowledge, reasoning, instruction-following, and safety tasks.

Measures expert-level understanding and reasoning across speech, environmental sound, and music.

Tests harder audio reasoning across speech, sound, music, spatial audio, multiple clips, voice chat, and instruction following.

Evaluates real-time voice agents on policy compliance, tool use, grounding, ambiguity, and state tracking.

Tests full-duplex voice agents on multi-tool tasks with human speech, disfluencies, interruptions, and latency measurements.

Extends τ-bench into spoken customer-service tasks to measure the capability gap between text and voice agents.

Separates enterprise task accuracy from conversational experience across realistic service scenarios.

Measures the silence callers experience between finishing a turn and hearing a voice-agent platform begin its reply.

A multilingual voice-agent suite covering tools, workflows, multi-turn interaction, and safety.

Measures how voice systems handle ambiguous, disfluent, and underspecified spoken dialogue.