Multilingual transcription and code-switching, explained

Most speech-to-text APIs were built for one language at a time. Real audio doesn't work that way. A support call switches between English and Spanish mid-sentence. A conference panel has speakers from six countries. A doctor dictates in German and drops in English drug names. If your transcription pipeline can't handle that, you're either running the wrong model per file or losing accuracy exactly where it matters most.

Multilingual transcription is the ability to turn speech into text across many languages — and, critically, to keep working when more than one language shows up in the same recording. This guide covers what that means, where it gets hard, and how we handle it in a single API call.

What is multilingual transcription?

Multilingual transcription is automatic speech recognition that spans more than one language. In practice, three separate capabilities get bundled under that one term, and it’s worth pulling them apart:

• Multi-language coverage — the model transcribes audio in any of the languages it supports, one language per file.
• Automatic language detection — you don't have to tell the API what language the audio is in; it figures that out for you.
• Code-switching — the hard one: a single speaker moves between two or more languages within the same utterance, and the model keeps up without dropping words or garbling the switch.

Coverage is table stakes now. Detection is common. Code-switching is where models separate — and it's the capability most teams underestimate until their real-world accuracy numbers come back lower than the demo promised.

The model: 99+ languages, 18 with native code-switching

Universal-3.5 Pro is our flagship async model at $0.21/hr. It transcribes across 99+ languages, and it handles native code-switching across 18 of them: English, Spanish, French, German, Italian, Portuguese, Arabic, Danish, Dutch, Finnish, Hebrew, Hindi, Japanese, Mandarin, Norwegian, Swedish, Turkish, and Vietnamese. That’s up from 6 code-switching languages in the prior generation — a 3x expansion — with no cost increase over Universal-3 Pro.

For the remaining languages, the model automatically falls back to Universal-2, which is how you get to the full 99+ count. You express that as an ordered fallback list, and it’s the default, so you don’t have to configure anything to get it:

import assemblyai as aai, os

aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],  # optional; this is the default
    language_detection=True,   # auto-detect the spoken language
    speaker_labels=True,
)

transcript = aai.Transcriber(config=config).transcribe(
    "https://assembly.ai/multilingual-call.mp3"
)

if transcript.status == aai.TranscriptStatus.error:
    raise RuntimeError(transcript.error)

print(transcript.text)

Two things to note. First, the parameter is speech_models — plural — because it's an ordered fallback list, not a single model id. Second, authentication uses your raw API key with no Bearer prefix; the SDK handles the header for you, but if you're calling the REST API directly, pass the key as-is.

Code-switching: the benchmark that matters

This is the section worth reading twice, because it's where the accuracy gap is widest. We measured normalized word error rate (WER) on code-switching audio — recordings where speakers move between languages mid-conversation — against the models teams actually evaluate us against. Lower is better.

Universal-3.5 Pro leads the field, and it does so by a real margin — that's a 22% relative reduction in code-switching WER over the prior generation, and it clears both ElevenLabs Scribe v2 and Deepgram Nova-3 ML. GPT-4o Transcribe, at 44.58, effectively falls apart on code-switched speech, which is a useful reminder that general-purpose transcription and purpose-built multilingual ASR are not the same product. You can see the full methodology on our benchmarks page.

Which speech-to-text API works best for multiple languages?

The honest answer is: benchmark it on your own audio, because “multilingual” means very different things depending on whose audio you feed it. But here’s how to reason about it.

If your audio is cleanly one language per file, most modern APIs will get you to a usable transcript, and you should compare on coverage, price, and diarization. If your audio has any code-switching — which it does more often than teams assume, especially in support, sales, and healthcare — that’s where you’ll see models diverge, and it’s the axis we optimize for hardest.

A couple of specific comparisons come up constantly:

• AssemblyAI vs Speechmatics for multilingual transcription: Speechmatics has strong language coverage; the differentiator is code-switching accuracy and the breadth of Speech Understanding features layered on top of the transcript. Run both on your worst code-switched files and compare WER directly.
• AssemblyAI vs ElevenLabs Scribe for transcription accuracy: Scribe v2 is a genuine contender and our closest competitor on the code-switching table above (8.77 vs our 7.69). We’d point you to the numbers and to prompting — you can bias our model with natural-language context and exact key terms in a way keyword-only approaches can’t match (more on that below).

For a structured way to run these comparisons, see our guide on how to evaluate speech recognition models.

Automatic language detection

You often don't know the language of an incoming file ahead of time — user-generated uploads, inbound calls, and media archives rarely come tagged. Set language_detection=True and the model identifies the spoken language for you, then transcribes it. If you do know the language, you can pin it with language_code to skip detection entirely. Both are optional; detection is the safe default for mixed or unknown inbound audio.

Diarization across languages

Multilingual audio is usually multi-speaker audio — a bilingual call has at least two people, a panel has several. Turn on speaker_labels=True and the model separates who said what. Universal-3.5 Pro ships the most accurate diarization we've released, optimized for concatenated minimum-permutation WER (cpWER), where it averages 30.17 versus Deepgram Nova-3 at 37.92 and ElevenLabs Scribe v2 at 35.26. That accuracy holds up when speakers switch languages, which is exactly when naive diarization tends to break.

Sharpening accuracy with prompting and key terms

Two tools help you push multilingual accuracy further, and they're complementary — not either/or:

• prompt is a short natural-language description of what the audio is about (2–50 words). It sets context — the domain, the scenario — so the model makes better guesses on ambiguous audio. It describes content, not formatting.
• keyterms_prompt is an explicit list of up to 1,000 words or phrases (max 6 words each) you want spelled correctly — product names, people, jargon that a general model wouldn't know.

Use them together: prompt to frame the conversation, keyterms_prompt to lock down exact spellings.

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    prompt="Bilingual customer support call in English and Spanish.",
    keyterms_prompt=["AssemblyAI", "Universal-3.5 Pro"],
)

Multilingual audio with PHI: Medical Mode and BAA

Healthcare audio is frequently multilingual, and it comes with an extra requirement: protected health information (PHI). Two things to know.

First, Medical Mode. Add domain: "medical-v1" (+$0.15/hr) and the model tunes for clinical vocabulary — drug names, procedures, and conditions that trip up general transcription. It reduces the missed clinical-entity rate by roughly 20% versus the base model, and it's available in English, Spanish, German, and French across both pre-recorded and streaming.

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro"],
    domain="medical-v1",
)

Second, compliance. AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process PHI. AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that safeguards PHI. You can also redact PHI directly with redact_pii across both audio and transcripts. Compliance details and the medical solution live on our site.

Who's using it

Ollang, which localizes media across languages, runs on AssemblyAI. Founder and CEO Ebru Yildirim: “The 30-40% reduction in speech-to-text errors has significantly improved our production efficiency and client satisfaction. We’ve achieved industry-leading word error rates for non-English audio, which is critical for serving our enterprise clients.” For multilingual teams, accuracy isn’t an edge case — it’s the default condition of their audio.

Multilingual transcription is available today through the Speech-to-Text API for pre-recorded audio and Streaming Speech-to-Text for real-time use. One caveat on streaming: on the Pro streaming model, language_detection isn’t supported — you bias to a language with language_code instead.

Get started

Explore Voice AI solutions to see multilingual transcription in your workflow, or get your free API key and run Universal-3.5 Pro on your own audio today.

Frequently asked questions

What's the best speech-to-text API for multiple languages?

Universal-3.5 Pro covers 99+ languages, with native code-switching across 18 of them, and leads the code-switching benchmark at 7.69 normalized WER. Benchmark it on your own audio — especially your hardest, most code-switched files — since that's where models diverge most.

Can AssemblyAI handle code-switching?

Yes. Universal-3.5 Pro handles native code-switching across 18 languages — a 3x expansion over the prior generation — and delivers a 22% relative reduction in code-switching WER versus Universal-3 Pro.

Can I automatically detect the language spoken in an audio file?

Yes. Set language_detection=True and the model identifies the language before transcribing. If you already know the language, pin it with language_code to skip detection.

Which languages support native code-switching?

English, Spanish, French, German, Italian, Portuguese, Arabic, Danish, Dutch, Finnish, Hebrew, Hindi, Japanese, Mandarin, Norwegian, Swedish, Turkish, and Vietnamese. For all other supported languages, the model falls back to Universal-2 for 99+ total coverage.

How accurate is multilingual transcription compared to a human?

On code-switched audio, Universal-3.5 Pro reaches 7.69 normalized WER — close to human transcription on clean speech and ahead of every competing model we benchmarked. Accuracy varies by language, audio quality, and domain, so test on representative audio.

Can I transcribe multilingual medical audio?

Yes. Add domain: "medical-v1" for clinical accuracy in English, Spanish, German, and French. For PHI, AssemblyAI is a business associate under HIPAA and offers a standard BAA, plus in-transcript PHI redaction.

‍