Top 8 speaker diarization libraries and APIs in 2026

Updated November 2026

Speaker diarization answers one deceptively simple question: who spoke when? Get it right and every downstream feature just works: per-speaker sentiment, agent-vs-customer analytics, speaker-attributed summaries. Get it wrong and all of that breaks, no matter how good your transcription is.

That's why diarization is one of the first things teams building conversation intelligence, voice agents and meeting products evaluate. This guide compares the eight diarization solutions worth knowing in 2026, shows how they score on the two metrics that matter, and separates the ones that run in real time from the batch-only options.

One framing note up front. Diarization is scored two ways, and they can rank the same systems differently. DER (diarization error rate) measures how much speech time lands on the wrong speaker. cpWER measures whether the right speaker label lands on the right words. We report both below, along with where AssemblyAI leads and where it doesn't.

What is speaker diarization?

Speaker diarization segments and labels an audio stream by speaker, so a transcript reads "Speaker A… Speaker B…" instead of one undifferentiated wall of text. In automatic speech recognition (ASR), it does two jobs: it works out how many distinct speakers are in a recording, and it attributes each segment of speech to the right one.

The big shift of the last couple of years is that diarization now runs in real time on a single live audio stream, not just on finished recordings. That's what makes live captioning, contact center coaching and voice agents possible without putting each speaker on a separate channel.

Async vs. streaming: the axis most teams overlook

Before the list, decide which mode you're building for, because it narrows your options fast.

Batch (async) diarization waits for the complete recording, then labels it. It can look at the whole file at once, so it's the most accurate option and the right choice for post-call analytics, meeting summaries and media transcription.

Streaming diarization assigns labels live, as people talk. It can't look ahead. It commits to a label with only the audio it has heard so far, so the first few turns are the least stable. But if speaker identity has to drive behavior during the conversation, like a voice agent telling the customer apart from a human agent, streaming is the only option.

Here's how the eight solutions compare. Benchmark figures come from the AssemblyAI benchmarks page: pre-recorded figures are from production APIs at default settings (October 2026), and realtime DER from the streaming diarization table (runs August 20 to September 24, 2026). Lower is better. A dash means the system has no run in that table.

Best speaker diarization APIs and libraries

1. AssemblyAI

AssemblyAI is a Voice AI platform built around speech-to-text, with Speech Understanding models for speaker identification, sentiment, entity detection and summarization layered on top.

For pre-recorded audio, Universal-3.5 Pro ranks first of nine systems on diarization error rate, at 12.2 against 14.6 for ElevenLabs Scribe v2 and 17.4 for Deepgram Nova-3. On cpWER it ranks second, at 26.7 against 26.4 for ElevenLabs Scribe v2. Both results are on the same four-dataset meeting and phone-call set (AMI, CallHome, DiPCo and NOTSOFAR), with DER also measured on VoxConverse and MSDWild.

For streaming, diarization runs on the same WebSocket as transcription with Universal-3.6 Pro Realtime. On realtime DER it ranks third of 12 systems at 16.68%. Meta's Muse Voice Transcribe leads at 14.51%, and Universal-3.5 Pro Realtime is second at 15.96%.

Turning it on takes one parameter. Here's the pre-recorded version with the Python SDK, using a speaker range:

from assemblyai import SpeakerOptions
from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

config = TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    speaker_labels=True,
    # Hard bounds on the number of labels. Use speakers_expected instead
    # only if you know the exact count. Never set both.
    speaker_options=SpeakerOptions(min_speakers_expected=2, max_speakers_expected=4),
)

transcriber = Transcriber(api_key="<YOUR_API_KEY>")
transcript = transcriber.transcribe("https://assembly.ai/wildfires.mp3", config)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

max_speakers_expected is a hard cap, not a hint. If more people speak than the cap, the extra voices merge into existing labels. If you leave it unset, the default depends on duration: 10 speakers for 2–10 minutes of audio and 30 for longer files. The speaker diarization docs recommend at least 30 seconds of speech per speaker for a stable label.

To swap "Speaker A" for real names or roles like "Agent" and "Customer", add Speaker Identification, which infers them from the conversation.

"Since moving to AssemblyAI, we've seen a meaningful improvement in the confidence tail of our production transcripts....What stands out is not just the model quality, but the way [they] let us bring real meeting context into transcription, from calendar titles to organizations, domains, and participant names, so recruiting conversations come through with the nuance our customers depend on." — Shahriar Tajbakhsh, Co-founder and CTO, Metaview

Key strengths:

• First on pre-recorded DER (12.2) and second on cpWER (26.7) on the benchmarks page
• Pre-recorded and streaming diarization on one API key
• Native code-switching across 18 languages on Universal-3.5 Pro and 32 on Universal-3.6 Pro Realtime, with Universal-2 for 99 languages
• Pricing: diarization adds $0.02/hr to pre-recorded transcription (Universal-3.5 Pro is $0.21/hr) and $0.12/hr to streaming (Universal-3.6 Pro Realtime is $0.45/hr). Speaker Identification starts at $0.02/hr.

2. Deepgram

Deepgram offers diarization on its Nova-3 model for both batch and streaming.

On the benchmarks page, Nova-3 averages 17.4 DER and 34.8 cpWER on pre-recorded audio, eighth and fifth of nine. In realtime, its DER is 34.01%, eleventh of 12 systems. Deepgram's API rate limits page caps streaming diarization at 50 concurrent requests in North America and 25 in Europe on Pay As You Go.

3. ElevenLabs Scribe

ElevenLabs Scribe v2 is the pre-recorded leader on cpWER, at 26.4, 0.3 ahead of Universal-3.5 Pro. Its DER is 14.6, second of nine. ElevenLabs' documentation lists support for up to 32 speakers.

Scribe v2 has no run in the realtime diarization table, so there's no published streaming comparison. Pick it for speaker-labeled transcripts of recorded audio if you're already on ElevenLabs. Check its realtime support against your requirements before building a live product on it.

4. Gladia

Gladia's Solaria-1 API offers speaker diarization on pre-recorded audio. On the benchmarks page it averages 16.7 DER and 36.5 cpWER, sixth and seventh of nine. Its documentation doesn't describe diarization for live streaming, so treat it as a batch option.

5. Speechmatics

Speechmatics offers diarization for batch and real-time transcription, with cloud and on-premise deployment. In real time you can set a maximum speaker count. On the benchmarks page, Speechmatics Enhanced averages 16.2 DER and 33.7 cpWER on pre-recorded audio, and its Enhanced Realtime model posts 22.16% realtime DER, sixth of 12. On-premise deployment is the main reason to shortlist it.

6. pyannote

pyannote is the most widely used open-source diarization toolkit. The current release line is pyannote.audio 4.x (4.0.7 on PyPI as of June 30, 2026), and its open pretrained pipeline is Community-1. You accept the model's conditions on Hugging Face, create an access token, and run it locally:

# pip install pyannote.audio
from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-community-1",
    token="{huggingface-token}")

output = pipeline("audio.wav")

for turn, speaker in output.speaker_diarization:
    print(f"{speaker} speaks between t={turn.start:.3f}s and t={turn.end:.3f}s")

pyannote returns speaker turns with timestamps, not a transcript. To get "who said what", you pair it with an ASR model and align the two yourself. It runs on CPU by default and needs a GPU for reasonable speed at volume. It's a fit for research teams and self-hosted builds with ML expertise to tune and maintain it.

7. NVIDIA NeMo

NeMo's Sortformer is an end-to-end neural diarization model that treats speaker labeling as one problem rather than a multi-stage pipeline, and NeMo integrates it with NVIDIA's ASR models. It's aimed at researchers and teams building custom multi-speaker ASR on their own GPUs.

8. Kaldi and SpeechBrain

Both are open-source research toolkits. Kaldi is the long-standing, highly configurable speech recognition framework, with x-vector and PLDA back ends for diarization: maximum control, steep learning curve. SpeechBrain is the PyTorch-based option, with a modular design and an active community. Both suit academic research and prototyping more than production, where self-hosting overhead dominates.

What about Whisper?

Whisper doesn't diarize. It returns text without speaker labels, so the common workaround is Whisper for transcription plus pyannote for speaker turns, aligned by timestamp. That works for prototypes, but alignment errors at turn boundaries and overlapping speech are yours to handle. Our guide to Whisper speaker diarization walks through the setup and its limits.

Real-time (streaming) speaker diarization

For a long time, reliable diarization meant batch processing. The only solid path to real-time speaker separation was multichannel audio with each speaker on a dedicated channel. That's no longer the case.

Streaming diarization now identifies who's speaking on a single live stream. With AssemblyAI you add speaker_labels: true to the streaming speech-to-text connection, plus an optional max_speakers cap (1–10) when you know the count. Labels arrive with each turn as people talk. When the session ends, a final refinement pass sends one SpeakerRevision message with only the turns whose labels changed. That pass adds about 400 ms, and the revision always arrives before the session's Termination message. For long sessions, speaker_labels_revision_interval_ms (300000 recommended) adds corrections while the stream is still running. Our post introducing SpeakerRevision explains the design.

Here's how the realtime diarization table on the benchmarks page looks across all 12 systems tested on CALLHOME, AMI, DiPCo and NOTSOFAR:

Reson8 caps speaker count at four, its product maximum, so its figures on NOTSOFAR (up to seven speakers) aren't directly comparable with systems that estimate speaker count freely.

A few practical notes for streaming:

• Short utterances are hard. Turns under about a second, like "yeah" or "uh huh", come back with the speaker label "PENDING", because the model needs more audio to attribute them. Handle that label in your client instead of assuming a letter.
• The first turns are the least stable. Early labels can be reassigned as the model hears more of each voice. Use the revised labels for anything you store or summarize.
• Fewer speakers means more accuracy. Accuracy is highest with two speakers and drops as the count climbs, since there's less audio per person.
• Multichannel is still great when your infrastructure already separates speakers onto channels, which is common in telephony. You get perfect separation with no diarization step.

How to evaluate speaker diarization quality

Most published diarization benchmarks report one number: diarization error rate (DER). It adds up missed speech, false alarms and speaker confusion as a share of total speech time. The standard setup uses a 0.25-second collar around reference boundaries and doesn't score overlapping speech.

DER is useful for isolating the diarizer, but it never looks at the words. A system can post a strong DER and still put the wrong words under the right label. cpWER fixes that. It computes word error rate after matching predicted speakers to reference speakers, so a correctly transcribed word on the wrong speaker counts as an error. For speaker-attributed transcripts, it's the closer proxy for what users see. Our post on DER vs. cpWER walks through the difference, and our guide to measuring speech-to-text accuracy covers both alongside WER and entity error.

The two metrics can disagree, so report both. On the pre-recorded tables, Universal-3.5 Pro is first on DER and second on cpWER. Grok STT is third on DER but sixth on cpWER.

For streaming, two more things matter:

• Latency: the delay between speech and a labeled turn. For a voice agent, a label that arrives after the agent has already replied is useless.
• Label stability: how often labels change. Constant flipping makes a UI look broken even when the final transcript is right. That's why AssemblyAI sends revisions as a separate message rather than relabeling turns in place.

Whatever you track, test on audio that matches your use case: turn-taking speed, audio quality, short utterances and overlapping speech.

How to choose a speaker diarization solution

For production applications:

• Lowest diarization error rate on recorded audio: AssemblyAI Universal-3.5 Pro (12.2 DER, first of nine).
• Lowest cpWER on recorded audio: ElevenLabs Scribe v2 (26.4), with Universal-3.5 Pro 0.3 behind at 26.7.
• Real-time voice agents and live coaching: a provider with production streaming diarization. AssemblyAI's Universal-3.5 Pro Realtime and Universal-3.6 Pro Realtime rank second and third of 12 on realtime DER, and diarization runs on the same connection as transcription.
• Contact centers: agent-vs-customer attribution for QA and coaching, as covered on our call analytics page.
• On-premise deployment: Speechmatics.

For research and development, the open-source options (pyannote 4.x with Community-1, NeMo's Sortformer, Kaldi, SpeechBrain) give you model control at the cost of engineering time, GPU infrastructure and ongoing maintenance. Many teams start on an API to validate the use case, then evaluate self-hosting once they understand their scale.

Frequently Asked Questions (FAQs)

What is the best speaker diarization API in 2026?

On AssemblyAI's published benchmarks, Universal-3.5 Pro has the lowest pre-recorded diarization error rate of nine systems at 12.2, ahead of ElevenLabs Scribe v2 at 14.6 and Deepgram Nova-3 at 17.4. On cpWER, ElevenLabs Scribe v2 is first at 26.4 and Universal-3.5 Pro second at 26.7. Choose on the metric that matches your product, and test on your own audio.

What is the best real-time speaker diarization API for voice agents?

You want streaming diarization on the same connection as transcription, with stable labels. On the realtime diarization table, Meta Muse Voice Transcribe leads at 14.51% DER, followed by AssemblyAI Universal-3.5 Pro Realtime (15.96%) and Universal-3.6 Pro Realtime (16.68%). Deepgram Nova-3 is at 34.01%.

What's the difference between DER and cpWER?

DER measures the share of speech time that is missed, falsely detected or attributed to the wrong speaker, and ignores the words. cpWER computes word error rate after matching predicted speakers to reference speakers, so a correct word on the wrong speaker counts as an error. DER isolates the diarizer, while cpWER reflects what users see in a speaker-labeled transcript.

How many speakers can speaker diarization detect?

It depends on the provider and mode. AssemblyAI's pre-recorded diarization defaults to a cap of 10 speakers for 2–10 minutes of audio and 30 for longer files, and you can set your own bounds with speakers_expected or speaker_options. Streaming diarization accepts a max_speakers cap from 1 to 10. Accuracy is highest with fewer speakers.

Is pyannote good enough for production?

It can be, if you have the engineering capacity. pyannote.audio 4.x with the Community-1 pipeline is free to self-host, but you run the GPUs, handle scaling, and align its speaker turns with a separate transcript yourself. Many teams start on an API and evaluate self-hosting once they know their volume.

How much does speaker diarization cost?

On AssemblyAI, diarization adds $0.02/hr to pre-recorded transcription and $0.12/hr to streaming, on top of the model rate ($0.21/hr for Universal-3.5 Pro, $0.45/hr for Universal-3.6 Pro Realtime). Speaker Identification, which replaces generic labels with names or roles, starts at $0.02/hr. Open-source libraries have no per-hour fee but carry GPU and engineering costs.