Speaker Diarization | AssemblyAI

Supported languages

Global Englishen

Australian Englishen_au

British Englishen_uk

US Englishen_us

Spanishes

Frenchfr

Germande

Italianit

Portuguesept

Dutchnl

Hindihi

Japaneseja

Chinesezh

Finnishfi

Koreanko

Polishpl

Russianru

Turkishtr

Ukrainianuk

Vietnamesevi

Afrikaansaf

Albaniansq

Amharicam

Arabicar

Armenianhy

Assameseas

Azerbaijaniaz

Bashkirba

Basqueeu

Belarusianbe

Bengalibn

Bosnianbs

Bretonbr

Bulgarianbg

Catalanca

Croatianhr

Czechcs

Danishda

Estonianet

Faroesefo

Galiciangl

Georgianka

Greekel

Gujaratigu

Haitianht

Hausaha

Hawaiianhaw

Hebrewhe

Hungarianhu

Icelandicis

Indonesianid

Javanesejw

Kannadakn

Kazakhkk

Laolo

Latinla

Latvianlv

Lingalaln

Lithuanianlt

Luxembourgishlb

Macedonianmk

Malagasymg

Malayms

Malayalamml

Maltesemt

Maorimi

Marathimr

Mongolianmn

Nepaline

Norwegianno

Norwegian Nynorsknn

Occitanoc

Panjabipa

Pashtops

Persianfa

Romanianro

Sanskritsa

Serbiansr

Shonasn

Sindhisd

Sinhalasi

Slovaksk

Sloveniansl

Somaliso

Sundanesesu

Swahilisw

Swedishsv

Tagalogtl

Tajiktg

Tamilta

Tatartt

Telugute

Turkmentk

Urduur

Uzbekuz

Welshcy

Yiddishyi

Yorubayo

Supported models

Slam-1slam-1

Universaluniversal

Supported regions

US & EU

The Speaker Diarization model lets you detect multiple speakers in an audio file and what each speaker said.

If you enable Speaker Diarization, the resulting transcript will return a list of utterances, where each utterance corresponds to an uninterrupted segment of speech from a single speaker.

Speaker Diarization and Multichannel

Speaker Diarization doesn’t support multichannel transcription. Enabling both Speaker Diarization and multichannel will result in an error.

Looking to identify speakers by name across multiple audio files? Check out our Speaker Identification guide to learn how to match speaker labels with actual speaker names.

Quickstart

Python SDK

Python

JavaScript SDK

JavaScript

C#

Ruby

PHP

To enable Speaker Diarization, set speaker_labels to True in the transcription config.

1 import assemblyai as aai
2 
3 aai.settings.api_key = "<YOUR_API_KEY>"
4 
5 # You can use a local filepath:
6 # audio_file = "./example.mp3"
7 
8 # Or use a publicly-accessible URL:
9 audio_file = (
10     "https://assembly.ai/wildfires.mp3"
11 )
12 
13 config = aai.TranscriptionConfig(
14   speaker_labels=True,
15 )
16 
17 transcript = aai.Transcriber().transcribe(audio_file, config)
18 
19 for utterance in transcript.utterances:
20   print(f"Speaker {utterance.speaker}: {utterance.text}")

Set number of speakers expected

You can set the number of speakers expected in the audio file by setting the speakers_expected parameter.

Only use this parameter if you are certain about the number of speakers in the audio file.

Python SDK

Python

JavaScript SDK

JavaScript

C#

Ruby

PHP

1 import assemblyai as aai
2 
3 aai.settings.api_key = "<YOUR_API_KEY>"
4 
5 # You can use a local filepath:
6 # audio_file = "./example.mp3"
7 
8 # Or use a publicly-accessible URL:
9 audio_file = (
10     "https://assembly.ai/wildfires.mp3"
11 )
12 
13 config = aai.TranscriptionConfig(
14   speaker_labels=True,
15   speakers_expected=5,
16 )
17 
18 transcript = aai.Transcriber().transcribe(audio_file, config)
19 
20 for utterance in transcript.utterances:
21   print(f"Speaker {utterance.speaker}: {utterance.text}")

Set a range of possible speakers

You can set a range of possible speakers in the audio file by setting the speaker_options parameter. By default, the model will return between 1 and 10 speakers.

This parameter is suitable for use cases where there is a known minimum/maximum number of speakers in the audio file that is outside the bounds of the default value of 1 to 10 speakers.

Setting max_speakers_expected too high may reduce diarization accuracy, causing sentences from the same speaker to be split across multiple speaker labels.

Python SDK

Python

JavaScript SDK

JavaScript

C#

Ruby

PHP

1 import assemblyai as aai
2 
3 aai.settings.api_key = "<YOUR_API_KEY>"
4 
5 # You can use a local filepath:
6 # audio_file = "./example.mp3"
7 
8 # Or use a publicly-accessible URL:
9 audio_file = (
10     "https://assembly.ai/wildfires.mp3"
11 )
12 
13 config = aai.TranscriptionConfig(
14   speaker_labels=True,
15   speaker_options=aai.SpeakerOptions(
16     min_speakers_expected=3,
17     max_speakers_expected=5
18   ),
19 )
20 
21 transcript = aai.Transcriber().transcribe(audio_file, config)
22 
23 for utterance in transcript.utterances:
24   print(f"Speaker {utterance.speaker}: {utterance.text}")

API reference

Request

Speakers Expected

$ curl https://api.assemblyai.com/v2/transcript \
> --header "Authorization: <YOUR_API_KEY>" \
> --header "Content-Type: application/json" \
> --data '{
>   "audio_url": "YOUR_AUDIO_URL",
>   "speaker_labels": true,
>   "speakers_expected": 3
> }'

Speaker Options

$ curl https://api.assemblyai.com/v2/transcript \
> --header "Authorization: <YOUR_API_KEY>" \
> --header "Content-Type: application/json" \
> --data '{
>   "audio_url": "YOUR_AUDIO_URL",
>   "speaker_labels": true,
>   "speaker_options": {
>     "min_speakers_expected": 3,
>     "max_speakers_expected": 5
>   }
> }'

Key	Type	Description
`speaker_labels`	boolean	Enable Speaker Diarization.
`speakers_expected`	number	Set number of speakers.
`speaker_options`	object	Set range of possible speakers.
`speaker_options.min_speakers_expected`	number	The minimum number of speakers expected in the audio file.
`speaker_options.max_speakers_expected`	number	The maximum number of speakers expected in the audio file.

Response

Key	Type	Description
`utterances`	array	A turn-by-turn temporal sequence of the transcript, where the i-th element is an object containing information about the i-th utterance in the audio file.
`utterances[i].confidence`	number	The confidence score for the transcript of this utterance.
`utterances[i].end`	number	The ending time, in milliseconds, of the utterance in the audio file.
`utterances[i].speaker`	string	The speaker of this utterance, where each speaker is assigned a sequential capital letter. For example, “A” for Speaker A, “B” for Speaker B, and so on.
`utterances[i].start`	number	The starting time, in milliseconds, of the utterance in the audio file.
`utterances[i].text`	string	The transcript for this utterance.
`utterances[i].words`	array	A sequential array for the words in the transcript, where the j-th element is an object containing information about the j-th word in the utterance.
`utterances[i].words[j].text`	string	The text of the j-th word in the i-th utterance.
`utterances[i].words[j].start`	number	The starting time for when the j-th word is spoken in the i-th utterance, in milliseconds.
`utterances[i].words[j].end`	number	The ending time for when the j-th word is spoken in the i-th utterance, in milliseconds.
`utterances[i].words[j].confidence`	number	The confidence score for the transcript of the j-th word in the i-th utterance.
`utterances[i].words[j].speaker`	string	The speaker who uttered the j-th word in the i-th utterance.

The response also includes the request parameters used to generate the transcript.

Frequently asked questions & troubleshooting

How can I improve the performance of the Speaker Diarization model?

To improve the performance of the Speaker Diarization model, it’s recommended to ensure that each speaker speaks for at least 30 seconds uninterrupted. Avoiding scenarios where a person only speaks a few short phrases like “Yeah”, “Right”, or “Sounds good” can also help. If possible, avoiding cross-talking can also improve performance.

How many speakers can the model handle?

By default, the upper limit on the number of speakers for Speaker Diarization is 10. If you expect more than 10 speakers, you can use speaker_options to set a range of possible speakers. Please note, setting max_speakers_expected too high may reduce diarization accuracy, causing sentences from the same speaker to be split across multiple speaker labels.

How accurate is the Speaker Diarization model?

The accuracy of the Speaker Diarization model depends on several factors, including the quality of the audio, the number of speakers, and the length of the audio file. Ensuring that each speaker speaks for at least 30 seconds uninterrupted and avoiding scenarios where a person only speaks a few short phrases can improve accuracy. However, it’s important to note that the model isn’t perfect and may make mistakes, especially in more challenging scenarios.

Why is the speaker diarization not performing as expected?

The speaker diarization may be performing poorly if a speaker only speaks once or infrequently throughout the audio file. Additionally, if the speaker speaks in short or single-word utterances, the model may struggle to create separate clusters for each speaker. Lastly, if the speakers sound similar, there may be difficulties in accurately identifying and separating them. Background noise, cross-talk, or an echo may also cause issues.

When should I use speakers_expected and when should I use speaker_options to set a range of speakers?

speakers_expected should be used only when you are confident that your audio file contains exactly the number of speakers you specify. If this number is incorrect, the diarization process, being forced to find an incorrect number of speakers, may produce random splits of single-speaker segments or merge multiple speakers into one in order to return the specified number of speakers. There are various scenarios where the audio file may include unexpected speakers, such as playback of recorded audio during a conversation or background speech from other people. To account for such cases, it is generally recommended to use min_speakers_expected instead of speakers_expected and to set max_speakers_expected slightly higher (e.g., min_speakers_expected + 2) to allow some flexibility.