Skip to main content
BenchLM
Data
Official provider results

SOV-Transcribe-1 launch evaluation

Salesforce reports a 5.44% mean word error rate across seven English Open ASR sets, 4.88% across six languages on the multilingual Open ASR sets and 8.21% on a 14-accent call-center set for SOV-Transcribe-1, plus a 76.33% task success rate on VoiceCodeBench. It quotes a 2.28% AA-WER Index from Artificial Analysis’s non-streaming speech-to-text board, first among open-weight models and seventh overall on November 12, 2026.

Source snapshots refreshed
Five measurement lanesVOICE / S2S

Reasoning, task completion, conversation dynamics, experience, and latency stay separate.

Back to directory

Benchmark profile

Scope
English short-form transcription (Open ASR sets), multilingual transcription in six European languages, accented call-center speech, entity-heavy dictation (VoiceCodeBench), contextual transcription with domain and keyword hints (ContextASR-Bench, Contextual Earnings-22), instruction-following transcription (Preference-ASR) and robustness (HALAS)
Primary metric
Word error rate (%, lower is better); VoiceCodeBench entity match and task success rate (%, higher is better)
Owner
Salesforce
Available evidence
Exact values in the Hugging Face model card’s results table (an image), with prompts and scoring code linked in its evaluation protocol

What it measures

Voice experience
Is the exchange natural, robust, and responsive?

Available results

Artificial Analysis and VoiceCodeBench

Artificial Analysis: word error rate (%, lower is better) on its non-streaming speech-to-text board, as Salesforce quotes it; the AA-WER Index combines the three sets, and the rank was read on November 12, 2026. VoiceCodeBench (BeSimple AI): canonical token and entity match and task success rate (%, higher is better), and word error rate (%, lower is better).

Artificial Analysis: word error rate (%, lower is better) on its non-streaming speech-to-text board, as Salesforce quotes it; the AA-WER Index combines the three sets, and the rank was read on November 12, 2026. VoiceCodeBench (BeSimple AI): canonical token and entity match and task success rate (%, higher is better), and word error rate (%, lower is better).
ModelAA-WER IndexAA-AgentTalkEarnings22-AAVoxPopuli-AAAA rank (open weights / overall)VoiceCodeBench entity matchVoiceCodeBench task successVoiceCodeBench WER
SOV-Transcribe-12.281.784.061.50#1 / #793.3976.335.01

English short-form transcription (Open ASR sets)

Salesforce-reported word error rate (%, lower is better), scored with the Open ASR Leaderboard normalizer. The public-set mean is Salesforce’s equal-weight mean of the seven sets listed. Earnings22 and VoxPopuli here are the Open ASR subsets, not the Artificial Analysis versions above.

Salesforce-reported word error rate (%, lower is better), scored with the Open ASR Leaderboard normalizer. The public-set mean is Salesforce’s equal-weight mean of the seven sets listed. Earnings22 and VoxPopuli here are the Open ASR subsets, not the Artificial Analysis versions above.
ModelPublic-set mean (7 sets)AMIEarnings22GigaSpeechMonsoon English (India)SPGISpeechURGENT24VoxPopuli
SOV-Transcribe-15.448.155.527.162.932.4110.471.46

Multilingual transcription (Open ASR sets)

Salesforce-reported word error rate (%, lower is better) on the Open ASR multilingual sets, with the Open ASR normalizer. The mean is the equal-weight mean of the six languages.

Salesforce-reported word error rate (%, lower is better) on the Open ASR multilingual sets, with the Open ASR normalizer. The mean is the equal-weight mean of the six languages.
ModelMean (6 languages)GermanFrenchItalianSpanishPortugueseDutch
SOV-Transcribe-14.884.055.194.373.434.597.67

Accents, context, instructions and robustness

Salesforce-reported word error rate (%, lower is better); Preference-ASR uses preference-aware WER. AppTek is the mean over 14 accents of call-center dialogues. ContextASR-Bench adds the domain (coarse) or the domain plus an entity list (fine) to the prompt; Contextual Earnings-22 adds a keyword list for the clip (local) or for the whole call, some absent from the clip (global). HALAS is a robustness set.

Salesforce-reported word error rate (%, lower is better); Preference-ASR uses preference-aware WER. AppTek is the mean over 14 accents of call-center dialogues. ContextASR-Bench adds the domain (coarse) or the domain plus an entity list (fine) to the prompt; Contextual Earnings-22 adds a keyword list for the clip (local) or for the whole call, some absent from the clip (global). HALAS is a robustness set.
ModelAppTek call-center (14 accents)ContextASR noneContextASR coarseContextASR fineContextual Earnings-22 noneContextual Earnings-22 localContextual Earnings-22 globalPreference-ASR defaultPreference-ASR instructedHALAS
SOV-Transcribe-18.217.037.103.6712.429.739.985.725.0525.69

Interpretation limit

Salesforce ran every result except the AA-WER rows, which it quotes from an independent speech-to-text board, using vLLM with greedy decoding, the DFlash draft off, a dataset-specific prompt for each set, and a repetition cleanup step adapted from the Qwen3-ASR toolkit. The prompts differ by set (verbatim with fillers for most English sets, fillers omitted for SPGISpeech, Latin script only for Monsoon English), so compare these numbers only with runs that used the same prompt and normalizer. The English public-set mean covers seven sets and leaves out LibriSpeech, so it is not the nine-set Open ASR mean other cards report. The multilingual mean covers German, French, Italian, Spanish, Portuguese and Dutch. The AA-WER figures are that board’s own measurements as Salesforce quotes them; the rank moves as the board adds systems. The card’s results table is an image; BenchLM transcribed Salesforce’s column only and does not carry the other models’ columns. These results do not enter text-model or voice-agent rankings.