Speaker diarization is the process of partitioning an audio recording by speaker identity: figuring out who spoke when, and attaching a label to every stretch of speech. It answers “who spoke when” without needing to know anyone’s name. The output turns a flat wall of transcript text into a readable conversation.
Speaker diarization in AssemblyAI: Set speaker_labels: true on a transcription request. The API returns a list of utterances, each tagged with a speaker label (A, B, C) and start and end timestamps. Universal-3.5 Pro, AssemblyAI’s pre-recorded flagship, posts an average cpWER of 30.17% across six public diarization datasets, the lowest of the 10 systems on our benchmarks page.
This guide covers how diarization works step by step, how it’s measured, how it differs on recordings vs. live streams, and how to add it to your product in a few lines of code.
How does speaker diarization work? The short answer
Most diarization systems follow four steps:
- Detect speech. Find the parts of the audio that contain a human voice.
- Segment. Cut that speech into chunks where only one person is talking.
- Embed. Turn each chunk into a vector, a numerical voiceprint.
- Cluster. Group similar voiceprints so each speaker gets one consistent label.
Modern models like Universal-3.5 Pro fold some of these steps together. They produce the transcript and the speaker changes jointly, instead of bolting a separate diarization pass onto the text afterward.
Why speaker diarization matters
Transcription tells you what was said. Diarization tells you who said it. Without attribution, most downstream analysis breaks.
- Contact centers need to separate agent from customer to score compliance, talk ratio, and sentiment per side.
- AI notetakers need to assign action items to the right person.
- Clinical documentation needs to know whether a symptom came from the patient or the clinician.
- Sales coaching measures who dominated the call and how often the rep interrupted.
Errors here cost more than they appear to. A single swapped label can flip who agreed to a price, or who reported a side effect. That’s why diarization accuracy is something product teams bill against, not a nice-to-have.
Sign up free and add speaker labels to your first transcript with one parameter. Diarization on Universal-3.5 Pro is a +$0.02/hr add-on.
The four steps in detail
1. Voice activity detection
Voice activity detection (VAD) strips out silence, music, hold tones, and background noise. Everything downstream works only on speech. Weak VAD lets noise leak into the speaker clusters, where it can show up as a phantom extra speaker.
2. Segmentation
Segmentation finds speaker-change points and cuts the speech into chunks that each hold one voice. Short turns are the hard part. A “yeah” or “mm-hmm” may be too brief to attribute with confidence, so systems handle back-channels carefully.
3. Embedding
Each segment is converted into a speaker embedding: a vector that captures pitch, timbre, and speaking style while ignoring the words themselves. Two segments from the same person should land close together in that vector space.
4. Clustering
Clustering groups the embeddings and decides how many speakers there are. Estimating the speaker count is one of the hardest problems in diarization. Too few clusters merges two people into one label. Too many splits one person across several. That’s why APIs let you pass an expected count or range when you know it.
Three architectural approaches
| Approach | How it works | Trade-off |
|---|---|---|
| Modular pipeline | Separate VAD, segmentation, embedding, and clustering models | Easy to debug and swap parts, but errors compound from stage to stage |
| End-to-end neural | One network maps audio frames directly to speaker activity | Handles overlapping speech better, but needs more training data |
| Hybrid / joint | Combines learned components, often jointly with transcription | The common production choice; balances accuracy and stability on messy audio |
Universal-3.5 Pro takes the joint route. It produces the transcript and the speaker changes together, which helps on short turns, rapid back-and-forth, and overlapped speech.
How diarization is measured: DER vs cpWER
Diarization error rate (DER) scores time: the share of audio seconds assigned to the wrong speaker, missed, or falsely detected. Concatenated minimum-permutation word error rate (cpWER) scores words: it measures whether each word in the transcript is both correct and attributed to the right speaker.
DER can rank systems backwards. A transcript can look broken to a reader and still post a good DER, and vice versa. Our research team walks through real examples in DER vs. cpWER: why the standard diarization metric ranks systems backwards. That’s why we optimize Universal-3.5 Pro for cpWER.
Average cpWER across DiPCo, CALLHOME (two splits), NOTSOFAR (two splits), and AMI (lower is better):
| Model | Average cpWER |
|---|---|
| AssemblyAI Universal-3.5 Pro | 30.17% |
| Azure | 30.35% |
| ElevenLabs Scribe v2 | 35.26% |
| Speechmatics | 36.60% |
| Gladia | 36.88% |
| Deepgram | 37.93% |
Source: AssemblyAI benchmarks, accessed October 2026. We publish where we lose, too. On realtime diarization, Universal-3.6 Pro Realtime posts a DER of 16.68%, third of 12 systems in our technical report (data as of September 16, 2026).
Diarization on recordings vs. live streams
Pre-recorded diarization sees the whole file before it labels anyone. It can compare every voice against every other voice, which makes it the more accurate setting.
Streaming diarization has to label each turn as it arrives. In AssemblyAI’s Streaming API, every Turn event carries a speaker_label, and every final word carries a speaker field. When the session ends, the server runs a refinement pass over the full conversation. It then sends a single SpeakerRevision message for any turns whose labels changed, adding roughly 400 ms. You can also request mid-session revisions for long calls.
For the full real-time picture, including the known limits on very short turns and cross-talk, read streaming speaker diarization: how to identify who’s speaking in real time.
How to add speaker diarization with AssemblyAI
Pre-recorded audio
import assemblyai as aai
aai.settings.api_key = "YOUR_API_KEY"
config = aai.TranscriptionConfig(
speech_models=["universal-3-5-pro"],
speaker_labels=True,
speakers_expected=2, # optional: only when you know the exact count
)
transcript = aai.Transcriber(config=config).transcribe("https://example.com/call.mp3")
for utterance in transcript.utterances:
print(f"Speaker {utterance.speaker}: {utterance.text}")Output:
Speaker A: Thanks for calling, how can I help?
Speaker B: Hi, I'm calling about my last invoice.If you don’t know the exact count, pass a range with speaker_options (min_speakers_expected and max_speakers_expected) instead. See the speaker diarization docs for defaults and limits.
Live streams
Add speaker_labels: true to your WebSocket connection parameters on Universal-3.6 Pro Realtime ("speech_model": "universal-3-6-pro"). Optionally cap speakers with max_speakers. Details are in the streaming diarization docs.
From “Speaker A” to real names
Diarization gives you anonymous labels. Speaker Identification maps them to names or roles, such as “Agent” and “Customer”, using the conversation’s content. No voice enrollment is required. It’s a +$0.02/hr Speech Understanding add-on.
| What you get from AssemblyAI | Model | Price |
|---|---|---|
| Pre-recorded speaker diarization | Universal-3.5 Pro, Universal-2 | +$0.02/hr (standard) |
| Streaming diarization with end-of-session revision | Universal-3.6 Pro Realtime | +$0.12/hr |
| Speaker Identification (names or roles) | Universal-3.5 Pro, Universal-2 | +$0.02/hr |
Base prices: Universal-3.5 Pro $0.21/hr; Universal-3.6 Pro Realtime $0.45/hr. Full details on the pricing page.
Try the playground and upload a two-speaker recording to see the utterances, labels, and timestamps on your own audio.
Where speaker diarization gets used
- Meetings and AI notetakers: attributed summaries and action items. See AI notetakers.
- Contact centers: agent vs. customer analytics, QA, and compliance. Read our conversation intelligence guide.
- Sales and recruiting: talk ratios, objection handling, and interview notes. Metaview saw roughly a 47% drop in low-confidence tokens after moving to Universal-3.5 Pro.
- Healthcare: separating clinician from patient in ambient documentation.
- Media: searchable speaker timelines for podcasts, interviews, and broadcasts.
- Voice agents: keeping the caller’s speech separate from other voices in the room.
Build it yourself or call an API?
Open-source toolkits like pyannote.audio and NVIDIA NeMo give you full control over the pipeline. They also give you the job of maintaining it: model updates, GPU capacity, speaker-count tuning, and pairing the diarization output with a separate ASR model’s timestamps.
An API returns transcript and speaker labels in one response, tuned and benchmarked for you. If you want to compare options side by side, see our roundup of the top speaker diarization libraries and APIs.
Where diarization is heading
Diarization is moving from a batch post-processing step to live infrastructure. Notetakers want labels during the meeting, not after it. Voice agents need to know which voice to answer. The next gains will come from joint models that treat “what was said” and “who said it” as one problem, plus better handling of overlap and very short turns. We cover those edge cases in the hard cases in speaker diarization.
Diarization also sits next to related tasks that people often confuse with it. Recognition and verification match voices to known identities; diarization never needs to. The diarization vs. recognition vs. identification breakdown explains the difference.
Try our API for free and get speaker-labeled transcripts from Universal-3.5 Pro, AssemblyAI's most accurate speaker diarization yet.
Frequently asked questions
What does “enable speaker diarization” mean?
It means turning on the setting that adds speaker labels to a transcript. In AssemblyAI’s API, you set speaker_labels: true. The response then returns utterances tagged Speaker A, Speaker B, and so on, each with start and end timestamps.
What is the best speaker diarization model?
Judge models by cpWER on audio like yours, because it measures words attributed to the right speaker. On AssemblyAI’s published benchmark, Universal-3.5 Pro has the lowest average cpWER (30.17%) of 10 systems, ahead of Azure (30.35%) and ElevenLabs Scribe v2 (35.26%). Always test on your own recordings.
Does Whisper support speaker diarization?
Open-source Whisper models don’t output speaker labels. Teams usually pair Whisper with a separate diarization toolkit such as pyannote, then align the two outputs by timestamp. APIs like AssemblyAI return the transcript and speaker labels together.
What is the difference between speaker separation and diarization?
Speaker separation splits overlapping voices into separate audio streams. Diarization labels who spoke when on the original audio without producing new audio. Separation can feed diarization, but most transcription use cases only need diarization.
What is the difference between speaker diarization and speaker recognition?
Diarization groups voices into anonymous labels without knowing who anyone is. Recognition matches a voice to a known, enrolled identity. AssemblyAI’s Speaker Identification is a third option: it maps diarized labels to names or roles from the conversation’s content, with no enrollment.
Can speaker diarization work in real time?
Yes. AssemblyAI’s Streaming API labels each turn live on Universal-3.6 Pro Realtime. At the end of the session, it sends one revision for any labels that changed, adding about 400 ms. Streaming diarization is a +$0.12/hr add-on.
