How to measure transcription accuracy and evaluate speech recognition models

To measure transcription accuracy, compare a model's transcript with a human-verified reference and calculate word error rate (WER): substitutions, deletions, and insertions divided by reference words. Then add entity accuracy, semantic WER, and cpWER for multi-speaker audio, and rerun the test automatically whenever your model, prompts, or audio change.

For years, evaluating speech-to-text models meant running WER against a clean dataset and declaring a winner. Simple. Reproducible. And increasingly wrong.

Here's the claim this guide backs up: the way most teams evaluate ASR in 2026 tells you little about how a model will behave in production. Not because the metrics are bad, but because they're incomplete, the reference transcripts underneath them are often broken, and most teams measure once instead of continuously.

Here's the story that convinced us. When we launched Universal-3 Pro in February 2026, a handful of customers reported that their internal benchmarks showed the new model performing worse than the one it replaced. Our own testing said the opposite. So we dug in, and what we found changed how we think about evaluation.

This guide walks through the core ASR evaluation metrics, the ground-truth problem, a practical benchmarking framework with working code, and how to automate the whole thing as a WER regression test in CI. For the results of these methods applied to current models, see how accurate speech-to-text is in 2026.

For reference, AssemblyAI's current models are Universal-3.5 Pro for pre-recorded audio (released July 7, 2026, $0.21/hr) and Universal-3.6 Pro Realtime for streaming (released September 29, 2026, $0.45/hr).

Part 1: The core ASR evaluation metrics

Word error rate: the foundation

Word error rate is still the industry standard, and it earned that spot. It's simple, reproducible, and gives you one number to compare across vendors. For a deeper primer, see Is word error rate useful?

WER is the percentage of words that differ between a reference transcript (your ground truth) and the model's output. It counts three error types:

• Insertions: words the model added that weren't spoken (hallucinations)
• Deletions: words that were spoken but the model missed
• Substitutions: words the model got wrong ("Cadillac" transcribed as "cataracts")

WER = (Insertions + Deletions + Substitutions) / Total words in reference × 100

A quick example:

• Reference: "The quick brown fox jumps over the lazy dog" (9 words)
• Output: "A quick brown fox jumped over the lazy dog"
• Errors: 2 substitutions ("The" → "A", "jumps" → "jumped")
• WER: (2 / 9) × 100 = 22.2%

The catch that trips up most teams: normalize both transcripts before you calculate anything. Lowercase everything, strip punctuation, spell out numbers, and standardize contractions. "don't" and "do not" mean the same thing but score as an error if you skip this. For English, use the open-source Whisper normalizer: the whisper-normalizer package on PyPI is a standalone port of OpenAI's English normalizer. Pin the version you benchmark with, because normalization rules change scores.

One more check before you trust any WER figure: how many datasets is it averaged over, and which ones? AssemblyAI's Universal-3.5 Pro averages 4.35% WER across four dataset groups on our benchmarks page: synthetic medical, accented English, general speech, and webinar speech. A figure from one curated dataset is marketing. A figure across several messy datasets is closer to a forecast.

Semantic WER: beyond word-for-word

Traditional WER assumes every word carries equal weight. But in modern Voice AI, transcripts usually feed an LLM, not a human reader. An LLM extracts the same meaning from "cannot" and "can't." Traditional WER penalizes that harmless difference as hard as a real error.

Semantic WER fixes this. A reasoning model receives the reference and the model transcript, judges whether they convey the same information, and scores on meaning rather than exact strings.

• Reference: "The patient is on hydrochlorothiazide for hypertension"
• Output: "The patient is on HCTZ for hypertension"
• Traditional WER: about 14% (one substitution in seven words)
• Semantic WER: 0% (HCTZ is the standard abbreviation)

For voice agents and LLM pipelines, semantic WER is often the more honest metric. Pipecat's open STT benchmark uses it to compare real-time providers.

Missed entity rate: the words that matter

WER treats "um" and "hydrochlorothiazide" as equals. Your business doesn't. Missed entity rate (MER) measures accuracy on high-value tokens: drug names and dosages, proper nouns, procedures, numbers, dates, addresses, and any domain terms you define. It isolates the words that decide whether your application works.

Medical transcription is the clearest example. AssemblyAI's Medical Mode is a +$0.15/hr add-on, enabled with domain: "medical-v1", that cuts missed medical entities by about 20% versus the base model. On AssemblyAI's benchmarks, it shows lower medical entity error than Deepgram, Speechmatics, AWS, and Google. AssemblyAI offers a Business Associate Addendum (BAA) for customers processing protected health information.

Character error rate (CER): when whole-word scoring falls apart

CER measures accuracy at the character level. It's the metric of choice when word boundaries are ambiguous, as in Mandarin, Japanese, Thai, and Korean. It's also useful for spelling-sensitive content like codes, IDs, and alphanumeric strings. Rule of thumb: use WER for space-delimited languages, CER when whole-word scoring stops making sense, and report both for multilingual models.

cpWER: how to measure diarization

A huge share of production audio has more than one speaker, and none of the metrics above tell you whether the model got the speakers right. You can score a perfect WER and still attribute the customer's words to the agent.

cpWER (concatenated minimum-permutation word error rate) fixes that. Concatenate each speaker's words, try every mapping between reference and predicted speakers, and keep the permutation with the lowest WER. One number then captures transcription and speaker attribution together. Our research team explains why cpWER ranks diarization systems more faithfully than DER. Universal-3.5 Pro's diarization was optimized directly for cpWER; see the head-to-head cpWER results.

Code-switching and multilingual accuracy

What happens when a speaker switches languages mid-sentence? "Necesito el report para el meeting de las tres" is normal for millions of bilingual speakers, and many models either force it into one language or garble the switch. If you serve multilingual users, build a code-switched test set from real audio and score it separately from your monolingual sets. A model can ace separate English and Spanish tests and still break on a mixed sentence.

LLM-as-a-judge

For a subjective quality read, use a reasoning model as a judge. Show it two anonymized transcripts plus the reference, ask which is better for your use case, and calculate win rates across a sample. We use current Claude models for this. It's slower and costlier than automated metrics, so use it to validate, not on every file.

Part 2: The ground truth problem

The biggest source of evaluation error usually isn't the model. It's your reference transcript.

Remember those Universal-3 Pro customers whose WER went up? When we pulled their reference files apart, the human transcripts were full of systematic errors, and the benchmark was penalizing the newer model for being more accurate than the humans. One example: the human reference said "Patient allergic to penicillium," and the model correctly wrote "penicillin." WER marked the model wrong.

Formatting choices make it worse: "healthcare" vs. "health care," "alright" vs. "all right," "12" vs. "twelve." Each swings WER without changing quality. We published the full investigation, including the semantic word lists we use to neutralize these, in why your WER benchmark might be lying to you. Read it before you run a single benchmark.

To clean a reference file, use AssemblyAI's free Truth File Corrector in the dashboard. It flags likely transcriber errors and semantic-equivalence issues and lets you approve each correction. In practice, this often shows that 5–15% of a "clean" reference file was inconsistency, not model error.

Part 3: A practical benchmarking framework

Step 1: Prepare representative audio

Pick 10 to 20 files that look like your real traffic: a mix of accents, a range of audio quality, real domain terms, and varied speaker counts with overlapping speech. Don't benchmark only on pristine audio. You want numbers that predict production, not numbers that flatter it.

Step 2: Build ground truth with quality controls

Option 1 (faster): transcribe with Universal-3.5 Pro, then have domain experts correct only clear errors. Option 2: hire professional transcribers and tell them to transcribe exactly what was said, not to clean up grammar. Either way, run the result through the Truth File Corrector.

Step 3: Normalize both transcripts

# pip install whisper-normalizer==0.1.15
from whisper_normalizer.english import EnglishTextNormalizer

normalizer = EnglishTextNormalizer()
reference_transcript = "Patient allergic to penicillin. Call 555-0100, OK?"
ai_transcript = "patient allergic to penicillin call 555-0100 ok"

reference_normalized = normalizer(reference_transcript)
ai_normalized = normalizer(ai_transcript)

Step 4: Calculate multiple metrics

Don't stop at WER. Get the error breakdown too, and add cpWER if your audio is multi-speaker. This uses jiwer 4.x, where process_words replaced the older compute_measures:

# pip install jiwer==4.0.0
import jiwer

result = jiwer.process_words(reference_normalized, ai_normalized)

print(f"WER: {result.wer:.2%}")
print(f"Substitutions: {result.substitutions}")
print(f"Deletions: {result.deletions}")
print(f"Insertions: {result.insertions}")

Step 5: Benchmark streaming and async separately

Pre-recorded models get time to process the whole file, so optimize purely for accuracy on final transcripts. Streaming models trade some headroom for speed, so measure time to first token, time to complete a turn, accuracy on final (not partial) transcripts, and deletion rate. A dropped word makes a voice agent's turn hang. Compare streaming accuracy at the latency you'll actually deploy. For voice agents, also test with the context you'll send in production: passing the agent's question via agent_context to Universal-3.6 Pro Realtime cut WER by 10.2% across 20,000 voice-agent files. Choosing an STT API for voice agents has the full framework.

Part 4: Automate it with WER regression testing in CI

Most teams evaluate once, pick a model, and never look again. That's risky, because the thing you measured keeps changing:

• Models change. If you omit speech_models, AssemblyAI routes requests to the newest Universal Pro model automatically. Providers also retire models: since October 6, 2026, Universal-2 requests in Chinese, Japanese, Swedish, and Hebrew are routed to Universal-3.5 Pro.
• Your config changes. New keyterms, prompts, or normalizer versions can move WER by points.
• Your audio changes. New customers, markets, and devices shift the mix.

So treat accuracy like any other test suite: a golden dataset, a pinned model, thresholds, a schedule, and alerts.

1. Build a golden set. Keep 100–300 representative files per audio slice (for example: phone calls, meetings, accented speech), each with a corrected reference transcript, in version control or object storage.
2. Record a baseline. Run the suite once and save WER per slice as your baseline.
3. Pin the model in tests. Set speech_models explicitly (for example ["universal-3-5-pro"]) so a run measures your config, not a silent upgrade. Test new models in a separate job before you switch.
4. Set thresholds. A sensible starting point: fail the run if any slice's WER rises more than 0.5 points and more than 5% relative to baseline. Requiring both keeps tiny moves on low-WER slices from failing the build. Before you trust any gate, run the suite three times with nothing changed and set it above the spread you see. In our test, the same audio on the same pinned model moved 1.6 points between runs on a small slice. Add the same gates for entity error rate on your critical terms, and cpWER if you diarize.
5. Schedule it. Run weekly or nightly, plus on any pull request that changes prompts, keyterms, or the evaluation set.
6. Alert and keep history. Post failures to Slack and store each run's results, so you can see drift over time instead of a single pass/fail.

Here's a minimal Python script that does steps 2–4. It transcribes each file with a pinned AssemblyAI model, normalizes, computes WER per slice, and exits non-zero on a regression. It also has an offline mode that scores saved transcripts, which is handy for testing thresholds.

import json, os, sys
from collections import defaultdict

import jiwer
from whisper_normalizer.english import EnglishTextNormalizer

MODEL = os.environ.get("AAI_SPEECH_MODEL", "universal-3-5-pro")  # pin the model you ship
MAX_ABS_INCREASE = 0.5   # fail only if a slice's WER rises more than 0.5 points...
MAX_REL_INCREASE = 0.05  # ...and more than 5% relative to its baseline

normalize = EnglishTextNormalizer()


def transcribe(audio_path):
    import assemblyai as aai
    aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]
    config = aai.TranscriptionConfig(speech_models=[MODEL])
    transcript = aai.Transcriber(config=config).transcribe(audio_path)
    if transcript.status == aai.TranscriptStatus.error:
        raise RuntimeError(f"{audio_path}: {transcript.error}")
    return transcript.text or ""


def main(manifest_path, baseline_path, hyp_dir=None):
    manifest = json.load(open(manifest_path))
    refs, hyps = defaultdict(list), defaultdict(list)
    for item in manifest:
        ref = open(item["reference"]).read()
        if hyp_dir:  # offline mode: score saved transcripts
            hyp = open(os.path.join(hyp_dir, item["id"] + ".txt")).read()
        else:
            hyp = transcribe(item["audio"])
        refs[item["slice"]].append(normalize(ref))
        hyps[item["slice"]].append(normalize(hyp))

    baseline = json.load(open(baseline_path))
    results, failures = {}, []
    for name in refs:
        wer = jiwer.wer(refs[name], hyps[name]) * 100
        results[name] = round(wer, 2)
        base = baseline.get(name)
        if base is None:
            continue
        if wer - base > MAX_ABS_INCREASE and wer > base * (1 + MAX_REL_INCREASE):
            failures.append(f"{name}: WER {wer:.2f}% vs baseline {base:.2f}%")

    json.dump({"model": MODEL, "wer": results}, open("wer_results.json", "w"), indent=2)
    print(json.dumps(results, indent=2))
    if failures:
        print("WER regression detected:\n" + "\n".join(failures))
        sys.exit(1)


if __name__ == "__main__":
    main(*sys.argv[1:])

The manifest lists each file, its reference transcript, and its slice:

[
  {"id": "call-001", "audio": "eval/audio/call-001.wav",
   "reference": "eval/refs/call-001.txt", "slice": "phone"},
  {"id": "mtg-001", "audio": "eval/audio/mtg-001.wav",
   "reference": "eval/refs/mtg-001.txt", "slice": "meetings"}
]

And a GitHub Actions workflow that runs it every Monday, on relevant pull requests, and on demand, then alerts Slack on failure:

name: wer-regression

on:
  schedule:
    - cron: "0 6 * * 1"   # every Monday at 06:00 UTC
  workflow_dispatch:
  pull_request:
    paths:
      - "prompts/**"
      - "keyterms/**"
      - "eval/**"

jobs:
  wer:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install assemblyai==1.6.1 jiwer==4.0.0 whisper-normalizer==0.1.15
      - name: Run WER regression
        env:
          ASSEMBLYAI_API_KEY: ${{ secrets.ASSEMBLYAI_API_KEY }}
          AAI_SPEECH_MODEL: universal-3-5-pro
        run: python wer_regression.py eval/manifest.json eval/baseline.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: wer-results
          path: wer_results.json
      - name: Alert Slack on failure
        if: failure()
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
        run: |
          curl -sS -X POST -H "Content-type: application/json" \
            --data "{\"text\": \"WER regression detected: $RUN_URL\"}" \
            "$SLACK_WEBHOOK_URL"

Want a fuller starting point with semantic word lists and corrected truth files? AssemblyAI's simple-semantic-wer-benchmark repo shows how to build an internal benchmarking pipeline. Clone it, add your API key to .env, and run python run_all.py to transcribe and score in one step.

Once a gate fails, don't just rerun it. Listen to the files that moved. Sometimes the model regressed; sometimes it's now right and your reference is wrong, and the Truth File Corrector is the next stop.

Part 5: The ASR evaluation metrics at a glance

My recommendation: for most teams, WER plus MER on your domain terms is the right primary pair, and both belong in your CI gates. Add semantic WER if transcripts feed an LLM, cpWER the moment you have more than one speaker, and code-switch WER if your users are multilingual.

Part 6: Real-world evaluation patterns

Medical transcription

MER on clinical terms is your north star, with WER and semantic WER as secondary checks. Clean your references first, and split streaming (ambient scribes) from async (batch documentation).

Voice agents

Weight semantic WER heavily and watch your deletion rate. Benchmark on real customer conversations with overlapping speech, test with and without agent context, and measure accuracy at your deployed latency.

Contact center analytics

Score WER on critical information (names, issues, account numbers) and cpWER for who said what. Use real calls with real background noise and accent diversity, never scrubbed samples.

Final words: evaluation is a system, not a step

Most teams treat evaluation as a gate they pass once before launch. It isn't. Your audio drifts, your use cases expand, models update, and your ground truth quietly rots. That's why the CI suite matters more than any single benchmark you run this quarter.

Here's the insight it took us a model launch and a lot of confused customers to learn: the metric you optimize is a decision about who your product serves. WER alone says every word matters equally. Adding cpWER says speaker attribution matters. Adding code-switch WER says bilingual users count. Adding semantic WER says you serve the LLM reading your transcript. Your metric stack is your product strategy, so choose it on purpose, then automate it.

Frequently asked questions

How do you measure transcription accuracy?

Compare the model's transcript with a human-verified reference after normalizing both, then calculate word error rate: (substitutions + deletions + insertions) / reference words. Add entity accuracy for critical terms and cpWER for multi-speaker audio. AssemblyAI recommends cleaning the reference first, because reference errors are the most common cause of misleading scores.

What metrics are used to evaluate an ASR model?

The core ASR evaluation metrics are word error rate (WER), semantic WER, missed entity rate (MER), character error rate (CER), and cpWER for diarization. LLM-as-a-judge adds a subjective quality read. For voice agents, add deletion rate and latency metrics like time to first token. AssemblyAI publishes WER, entity error, and cpWER results on its benchmarks page.

What's the difference between WER, semantic WER, and cpWER?

WER counts word-level errors against a reference. Semantic WER asks a reasoning model whether the meaning survived, so "can't" vs. "cannot" isn't an error. cpWER scores transcription and speaker attribution together, so a correct sentence given to the wrong speaker still counts. AssemblyAI optimized Universal-3.5 Pro's diarization directly for cpWER.

How do I automatically validate transcription accuracy?

Keep a golden set of representative audio with corrected reference transcripts, pin the model version, and run a script that computes WER per audio slice on a schedule in CI. Fail the run when WER rises past a threshold and alert your team. AssemblyAI's example above uses the Python SDK, jiwer, and GitHub Actions.

What WER increase should trigger a regression alert?

A practical starting point is to fail when a slice's WER rises more than 0.5 points and more than 5% relative to its baseline. Requiring both stops small swings on low-WER slices from failing the build. Use tighter gates for entity error rate on critical terms like account numbers or drug names. Tune thresholds after a few runs, once you know your normal run-to-run noise.

How often should I re-benchmark my speech-to-text provider?

Run your regression suite at least weekly, and rerun the full comparison whenever a provider ships a new model or you change prompts, keyterms, or audio sources. AssemblyAI shipped Universal-3.5 Pro in July 2026 and Universal-3.6 Pro Realtime in September 2026, so a quarterly-or-faster cadence keeps you current.