Now live New models for transcription and text-to-speech. AssemblyAI-STT-3.7 AssemblyAI-TTS-1
Insights & Use Cases

The hard cases in speaker diarization: overlap, short turns, and noise

Diarization looks solved on a clean call and falls apart at a dinner party. Here's what breaks it (overlap, short clips, short turns, noise) and how to measure it.

Abstract green cylinder illustration

Written by

Kelsey Foster

Published on

19 November 2026

Speaker diarization looks solved right up until it isn't. Point a decent model at a clean two-person phone call and it'll nail who said what, turn after turn, with almost no effort. That's the demo everyone shows.

Now drop the same model into a dinner party. Four people, a cross-talk moment, somebody laughing mid-sentence, a "yeah, exactly" fired off while the host is still finishing a thought. Or hand it a 40-second voice note. The clean-call model falls apart, and it falls apart in specific, nameable ways.

This post is that list: the cases that break diarization systems, why each one breaks them, and how you'd know it's failing. That last part matters more than it sounds, because the metric most people quote barely notices some of the worst failures. Along the way you'll see what changed with the new speaker diarization model behind AssemblyAI-STT-3.7, which made its biggest gains on exactly these cases.

Why diarization is still hard

Diarization is two jobs stapled together. First, figure out how many distinct speakers exist. Second, draw the boundaries that say person A talked from here to here, then person B took over. Do both perfectly and you get a clean speaker-labeled transcript. Miss on either and the errors compound.

On a clean call, both jobs are easy. Two clearly different voices, one talking at a time, generous pauses between turns: the acoustic signal practically labels itself. If you want the full mechanics of how this works under the hood, we wrote a complete explainer on speaker diarization that walks through the pipeline.

Real conversations don't cooperate. People interrupt. They react. They talk over each other. They mumble from across a room. They send a voice note that's over before the model has heard enough of anyone. When reality gets messy, the failures sort into a handful of recurring categories.

Overlapped speech

Two people talking at once is the single biggest source of diarization error.

Here's why it's so brutal. Most diarization systems are built on an assumption baked deep into their architecture: at any given instant, exactly one person is speaking. That assumption makes the math tractable. It also makes overlap a blind spot, because when two voices land in the same slice of audio, the system has to pick one, and the other speaker's words can vanish.

Some systems handle overlap by masking it, detecting the overlapped region and excluding it from analysis so it doesn't corrupt the clustering. That keeps the speaker labels cleaner. It also throws away the audio where two people were talking. For a meeting-summarization use case or a call-center QA workflow, those are often the words that matter most: the interruption, the objection, the correction.

Good overlap handling attributes overlapping speech to the right speakers and keeps both sets of words. That's a real bar, and most legacy pipelines don't clear it.

How do speech services handle overlapping speech in transcripts?

When two people talk at once, a transcription system has to decide what the text looks like. There are four common approaches, and each trades something away.

Approach What the transcript looks like What you lose
One speaker per segment The overlapped stretch goes to whoever the model picks The other speaker's interjection gets merged into the wrong turn or dropped
Mask the overlap Overlapped regions are excluded from speaker clustering Cleaner clusters, but the words in that region can go missing
Word-level attribution in a serialized transcript One ordered list of utterances; each word carries one speaker label Truly simultaneous words still have to be ordered in one stream
Separate channels Each participant is transcribed from their own channel Nothing on attribution, but the recording setup has to capture it

AssemblyAI's pre-recorded API uses the third approach on mono audio. With speaker_labels: true, the response includes an utterances array. Each utterance is an uninterrupted stretch from one speaker, and every word inside it has its own speaker, start and end fields. The new diarization model weighs its confidence for each word when it assigns a speaker. Full sentences stay with one speaker, and an interjection during someone else's sentence comes back as its own short utterance under the right label.

When you need both voices in full during heavy crosstalk, the fix is in the recording setup. If each participant is on their own channel, such as the two legs of a phone call or separate mics on a podcast, set multichannel: true. Each channel is transcribed on its own, so nothing has to be inferred about who spoke. You can combine multichannel with speaker_labels when a channel holds more than one person; labels then come back as 1A, 1B and 2A. The tradeoffs are covered in multichannel speaker diarization.

Short turns and back-channels

"Yep." "No." "Right." A quick laugh. A "wait, what?" thrown in while someone else holds the floor.

These are back-channels, the tiny verbal reactions that keep a conversation moving, and they wreck diarization in a sneaky way. The system credits them to whoever was already speaking. Person A is mid-sentence, Person B says "exactly," and the transcript hands that "exactly" to Person A. The turn was too short and too embedded to trigger a speaker switch, so it gets swallowed.

Now here's the part that should change how you evaluate diarization. You might never see this failure in your metrics.

The standard metric, DER (diarization error rate), weights errors by seconds of audio. A back-channel is maybe 300 milliseconds. Misattribute it and DER barely moves. So you can have a system that misfiles every short reaction in your data and still posts a DER that looks great on a slide.

We ran the numbers on this and the gap is almost comical. In our own testing, a catastrophic attribution output, with words credited to the wrong people all over the place, scored just 15.1% DER. The same output scored 30.7% cpWER, twice as bad by the metric that tracks whether each person's words landed in the right place.

It cuts the other way too. A flawless transcript, every word right and every word attributed correctly, scored 51.5% DER but 0.0% cpWER. DER called a perfect result a failure. If you're optimizing for DER, you're optimizing for the wrong thing, and short turns are exactly where that bites. We broke this whole comparison down in DER vs cpWER if you want the full teardown.

Test Your Messiest Audio

Short turns and cross-talk are where diarization fails without warning. Drop an interview with interruptions or a one-minute voice note into the playground and see whether the "exactly" lands on the right speaker.

Try playground

Short clips and voice notes

Clustering needs evidence. Each speaker's voice embedding gets steadier the more audio the model hears from that person, which is why the Speaker Diarization docs recommend at least 30 seconds of speech per speaker for a stable label.

A short clip breaks that budget before anyone says anything unusual. A 40-second voice note with two people in it gives each speaker maybe 15 seconds of audio. Three things go wrong:

  • The speaker count drifts. With little audio, one person whose voice shifts mid-sentence can look like two people, or two similar voices can look like one.
  • Short turns get absorbed. On a long recording the model has heard each voice plenty of times elsewhere. On a 40-second clip it may have heard the person who said "mm-hmm" only once.
  • Labels don't carry across files. Diarization labels are per file. "Speaker A" in one voice note has nothing to do with "Speaker A" in the next, so chopping a conversation into snippets gives each snippet less audio and resets the labels every time.

That's why voice notes, short call snippets, clipped highlights and segmented meeting recordings have been where speaker labels looked worst. The AssemblyAI-STT-3.7 research post breaks diarization results out by clip length. It compares Universal-3.5 Pro with the previous diarization model, Universal-3.5 Pro with the new one, and AssemblyAI-STT-3.7 with the new one:

Metric and clip length Universal-3.5 Pro + previous diarization Universal-3.5 Pro + new diarization AssemblyAI-STT-3.7 + new diarization
cpWER, 60-second clips 34.0% 22.5% 18.6%
cpWER, 120-second clips 31.5% 22.2% 18.2%
DER, 60-second clips 25.4% 19.5% 17.5%
DER, 120-second clips 23.0% 18.4% 16.5%
Speaker-count difference, 60-second clips 0.91 0.39 0.39
Speaker-count difference, 120-second clips 0.70 0.29 0.28

Averages over English speaker-annotated conversation datasets. The clip groups use the two datasets every system ran as clips, so they aren't directly comparable with full-recording results. Speaker-count difference is predicted minus true speakers, averaged over files.

Read the speaker-count rows first. On 60-second clips, the previous system was off by almost a full speaker per file on average; the new model cuts that to 0.39, so most clips now come back with the right number of people. cpWER fell by nearly half on the same clips. Most of that gain arrives with the diarization model itself, which is now rolling out across all AssemblyAI models and every supported language.

Background noise and far-field audio

Not every conversation happens through a headset an inch from someone's mouth. A lot of the audio that matters most is captured in the worst possible conditions.

Think about a drive-thru: engine noise, wind, a speaker box, someone in the back seat. Or a self-service kiosk in a busy lobby. Or a conference room where the mic is in the center of the table and half the speakers are eight feet away. Or a phone on speaker sitting on a desk. This is far-field audio, and it stacks two problems at once.

First, the target voices are quieter and smeared by room reverberation, which blurs the acoustic fingerprint the system uses to tell speakers apart. Second, background noise and background speech (a TV, a nearby conversation, ambient chatter) give the model extra "voices" to trip over. It can invent speakers that don't exist or fail to separate the ones that do.

Most of the fix happens at capture. Close mics beat room mics, separate channels beat a single mixed track, and keeping a phone leg and a studio mic apart beats mixing them into one mono file. A single-mic room recording and a per-speaker channel setup call for different strategies, and the second one sidesteps most of the far-field problem by design.

Getting the speaker count wrong

Every diarization output has a hidden claim baked into it: this conversation had N speakers. Get N wrong and everything downstream is wrong too.

There are two ways to blow it. Merging is when the system collapses two people into one: two similar-sounding colleagues become "Speaker A," and now half of one person's words are filed under the other. Splitting is the opposite: one person's voice shifts (they get excited, they lower their voice, the line quality changes) and the system decides that's a new person, so "Speaker A" and "Speaker C" are the same human.

Similar-sounding voices make merging worse. Two men with comparable pitch and cadence, or two people with the same regional accent, sit close together in the model's internal space, and the clustering can't pull them apart.

That's why you need a roster metric, speaker count error, that asks a blunt question: did the system get the number of people right? It's separate from word-level accuracy, and it catches structural failures that a per-word score can smear across the whole transcript. It's also where the new diarization model moved most: across full recordings, the average speaker-count difference fell from 0.57 to 0.21.

When you know how many people are on the recording, say so. Use speakers_expected for an exact count, or speaker_options with min_speakers_expected and max_speakers_expected for a range. Both are hard limits: extra voices get merged into existing labels, and a maximum set too high can over-split. The full set of tuning moves, including Speaker Identification for names and roles, is in how to improve speaker diarization accuracy.

How to tell if your diarization is failing

By now the theme should be clear. The failures that hurt most (short turns, misattribution, cross-talk) are the ones DER is worst at seeing. If you only track DER, you're flying blind on your hardest cases.

Here's the evaluation stack that works.

cpWER (concatenated minimum-permutation word error rate). For each person in the conversation, what fraction of their words did the system get wrong: misrecognized, dropped, or credited to someone else? You concatenate everything each speaker said, find the speaker mapping that gives the best possible match, and score the errors. Because it's grounded in words instead of seconds, a misattributed "exactly" costs exactly as much as it should.

Speaker count error. The roster metric from the last section. Track it alongside cpWER so structural merges and splits can't hide.

Speaker confidence, in production. Both metrics need a labeled reference. On live traffic you don't have one, so ask the model where it was unsure. Set include_speaker_confidence: true inside speaker_options and every word and utterance comes back with a speaker_confidence score. Low scores cluster around crosstalk and very short turns, which makes them a natural review queue.

How AssemblyAI-STT-3.7 handles the hard cases

Most diarization systems decide who is speaking in each stretch of audio, then give every word in that stretch the same label, even where the model wasn't sure. The new speaker diarization model behind AssemblyAI-STT-3.7 weighs its confidence for each word when it assigns a speaker. Sentences stay intact, and quick interjections like "yeah" and "mm-hmm" still go to the person who said them.

According to the AssemblyAI-STT-3.7 launch post, AssemblyAI-STT-3.7 has the lowest diarization error rate of the pre-recorded APIs measured: 10.7%, averaged over six English conversation datasets. That's ahead of ElevenLabs Scribe v3 at 12.0%. No other provider came in below 15%, a group that includes Speechmatics Enhanced, Gladia Solaria-1, AWS Transcribe and Deepgram Nova-3. Against Universal-3.5 Pro at its launch, cpWER is down 20%, DER down 36% and speaker-count errors down 63%, and the diarization step runs 3–4× faster.

If you already send speaker_labels: true, the new model is live for every language on all of AssemblyAI's pre-recorded speech models with no code change. To run it on AssemblyAI-STT-3.7 with the Python SDK:

import os

from assemblyai.prerecorded.v2 import Transcriber, TranscriptionConfig

transcriber = Transcriber(api_key=os.environ["ASSEMBLYAI_API_KEY"])
config = TranscriptionConfig(speech_models=["assemblyai-stt-3-7"], speaker_labels=True)

transcript = transcriber.transcribe("./meeting.wav", config=config)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

Real-time is where diarization gets thornier, because you can't cluster speakers you haven't heard yet. AssemblyAI-STT-3.7-Realtime makes speaker labels final the moment a turn ends, nearly as accurate as labels from pre-recorded audio, with no revision pass to wait for. It has the lowest live diarization error rate of every realtime provider measured on telephone calls and on multi-party meetings recorded on a single distant microphone, ahead of ElevenLabs Scribe v3 Live and Meta Muse Voice Transcribe. You can cap the roster with max_speakers (1–10). The mechanics are in streaming speaker diarization.

Code: flag the speaker labels you shouldn't trust

The script below transcribes a file on AssemblyAI-STT-3.7 with speaker labels and confidence scores. It then flags four things:

  • speakers with under 30 seconds of speech
  • turns shorter than one second
  • places where consecutive speakers' timestamps overlap
  • words whose speaker_confidence falls below a threshold

It calls the REST API with requests, matching the Python example in the docs; the Python SDK's SpeakerOptions doesn't expose include_speaker_confidence yet. The full version, with command-line flags and multichannel support, is in the companion repo on GitHub.

pip install requests
export ASSEMBLYAI_API_KEY="your-api-key"
import os
import sys
import time
from collections import defaultdict

import requests

BASE_URL = "https://api.assemblyai.com"
HEADERS = {"authorization": os.environ["ASSEMBLYAI_API_KEY"]}
MIN_SPEECH_PER_SPEAKER_MS = 30_000  # docs: at least 30 s of speech per speaker
SHORT_TURN_MS = 1_000
CONFIDENCE_THRESHOLD = 0.5  # tune on your own audio


def upload(path):
    with open(path, "rb") as f:
        response = requests.post(f"{BASE_URL}/v2/upload", headers=HEADERS, data=f)
    response.raise_for_status()
    return response.json()["upload_url"]


def transcribe(audio_url, num_speakers=None):
    speaker_options = {"include_speaker_confidence": True}
    if num_speakers:
        # speakers_expected can't be combined with speaker_options,
        # so pin the count with an equal min and max instead.
        speaker_options["min_speakers_expected"] = num_speakers
        speaker_options["max_speakers_expected"] = num_speakers

    body = {
        "audio_url": audio_url,
        "speech_models": ["assemblyai-stt-3-7"],
        "speaker_labels": True,
        "speaker_options": speaker_options,
    }
    response = requests.post(f"{BASE_URL}/v2/transcript", headers=HEADERS, json=body)
    response.raise_for_status()
    polling_url = f"{BASE_URL}/v2/transcript/{response.json()['id']}"

    while True:
        result = requests.get(polling_url, headers=HEADERS).json()
        if result["status"] == "completed":
            return result
        if result["status"] == "error":
            raise RuntimeError(f"Transcription failed: {result['error']}")
        time.sleep(3)


def report(transcript):
    utterances = transcript["utterances"] or []

    talk_ms = defaultdict(int)
    for u in utterances:
        talk_ms[u["speaker"]] += u["end"] - u["start"]  # start/end are in ms
    for speaker, ms in sorted(talk_ms.items()):
        note = "  <- under 30 s, label may be unstable" if ms < MIN_SPEECH_PER_SPEAKER_MS else ""
        print(f"Speaker {speaker}: {ms / 1000:.1f}s of speech{note}")

    for u in utterances:
        if u["end"] - u["start"] < SHORT_TURN_MS:
            print(f"Short turn at {u['start'] / 1000:.2f}s, {u['speaker']}: {u['text']!r}")

    for a, b in zip(utterances, utterances[1:]):
        if a["speaker"] != b["speaker"] and b["start"] < a["end"]:
            print(f"Overlap at {b['start'] / 1000:.2f}s: {a['speaker']} -> {b['speaker']}")

    for u in utterances:
        for w in u["words"]:
            score = w.get("speaker_confidence")
            if score is not None and score < CONFIDENCE_THRESHOLD:
                print(f"Low speaker confidence at {w['start'] / 1000:.2f}s, "
                      f"{w['speaker']}: {w['text']!r} ({score:.2f})")


if __name__ == "__main__":
    audio = sys.argv[1]
    speakers = int(sys.argv[2]) if len(sys.argv) > 2 else None
    audio_url = audio if audio.startswith("http") else upload(audio)
    transcript = transcribe(audio_url, speakers)

    print(f"Model: {transcript['speech_model_used']}, duration: {transcript['audio_duration']}s\n")
    for u in transcript["utterances"] or []:
        print(f"Speaker {u['speaker']}: {u['text']}")
    print()
    report(transcript)

Run it on a voice note, passing the speaker count as an optional second argument: python check_speaker_labels.py voice-note.m4a 2.

Use the flags as a review queue. A low-confidence word inside a one-second turn that overlaps the previous speaker is the most likely misattribution in the file. Send those spans to a human reviewer, or show them in a softer style in your UI. A short "yes" with high confidence is usually right, so a flag on its own isn't an error. And note the units: audio_duration is in seconds, while utterance and word start and end are in milliseconds.

Get Speaker Labels You Can Build On

Run AssemblyAI-STT-3.7 with speaker labels and confidence scores on your own recordings. New accounts get $50 in free credits for the speech APIs.

Sign up free

What the hard cases look like now

Short audio used to be where the practical advice was to skip speaker labels or batch clips into longer files first. With the speaker count right on most 60-second clips and cpWER nearly halved, a voice-memo app, a clip-sharing feature or a QA tool that samples 90 seconds of each call can show who said what directly.

Overlap is a different kind of problem. A single mono stream can only record that two people spoke at once; it can't keep both voices fully apart. Better word-level attribution reduces the damage, and confidence scores show you where the remaining doubt sits. Where crosstalk is the norm, as in debates, group calls and family dinners, separate channels still win.

For the full spec, see the speaker diarization feature page and the pre-recorded Speech-to-Text API. AssemblyAI-STT-3.7 is $0.21/hr, and standard speaker diarization adds $0.02/hr.

See How It Handles Your Hard Cases

Overlap, short clips, back-channels and far-field audio: run your worst files through the new diarization model. Start with $50 in free credits for the speech APIs.

Sign up free

Frequently Asked Questions (FAQs)

What are the limitations of speaker diarization?

The main limitations of speaker diarization are overlapping speech, short clips, short turns and back-channels, background noise and far-field audio, speaker-count errors, and similar-sounding voices. Overlapping speech is the single biggest source of error, because most systems assume one person talks at a time. Short reactions like "yep" get credited to the wrong speaker, and short recordings give the model too little audio per voice. Models that assign speakers word by word, using their own confidence, handle these cases far better than segment-level pipelines.

What is overlapping speech?

Overlapping speech is when two or more people talk at the same time, so their voices land in the same slice of audio. It's the top source of diarization error because most systems attribute each moment to a single speaker, which forces them to pick one voice. Systems that mask overlap keep cleaner speaker labels but can drop the words in the overlapped region, often the interruptions and corrections that matter most.

Why does speaker diarization fail on short audio clips?

Speaker diarization fails on short clips because the model hears too little of each voice to build a stable voice embedding, so it miscounts speakers and misattributes short turns. AssemblyAI's docs recommend at least 30 seconds of speech per speaker. On 60-second clips, AssemblyAI's new diarization model cut the average speaker-count error from 0.91 to 0.39 and cpWER from 34.0% to 18.6%.

How does background noise affect speaker diarization?

Background noise degrades diarization by masking the acoustic features a system uses to tell speakers apart and by adding stray sounds it can mistake for extra speakers. Far-field audio (kiosks, drive-thrus, conference rooms, speakerphones) makes it worse, since distance and reverberation blur each voice's fingerprint. Close microphones and a separate channel per speaker are the most reliable fixes.

Why do diarization systems get the number of speakers wrong?

Diarization systems miscount speakers in two ways: merging and splitting. Merging collapses two similar-sounding people into one speaker, while splitting treats one person's changing voice as a brand-new speaker. Tracking speaker count error as a dedicated roster metric catches these structural mistakes, and passing an expected speaker count or range reduces them.

How do you measure speaker diarization accuracy?

Measure diarization accuracy with cpWER (concatenated minimum-permutation word error rate) plus speaker count error, not DER alone. cpWER checks, for each person, what fraction of their words the system got wrong, so it catches short-turn misattribution that DER's seconds-based weighting misses. In our testing a catastrophic attribution output scored just 15.1% DER but 30.7% cpWER, while a flawless transcript scored 51.5% DER yet 0.0% cpWER.