Voice cloning

​

Create a voice

• One person, speaking the voice’s language. Pick the language in the dialog and have the speaker read in it: a speaker reading English into a voice set to French is cloned on the wrong sounds.
• Ten to thirty seconds of speech. The dashboard uses up to thirty seconds of a recording under two minutes long, and shows when a recording is long enough. Read the script it offers, calm or excited, or your own text at a natural pace.
• Consent. You attest that the speaker consented to the clone, and your account must have accepted the biometric data terms on the dashboard’s data controls page.

​

Get the best clone

• Use a good microphone, close to the mouth. The closer the microphone, the more of the voice it picks up. A headset or earbuds, such as AirPods, work well.
• Record somewhere quiet, with little echo. Background noise and room echo come through as artifacts in the generated speech. A small room with soft furnishings echoes less than a large, bare one.
• Upload a WAV file. An uncompressed file keeps detail that MP3 and other compressed formats lose.
• Speak in the tone and pitch you want to hear. The voice speaks the way the recording does, so record calm for a calm voice and upbeat for an upbeat one.
• Keep the quirks you want in the voice. Raspiness, filler words and hesitations, and the way the speaker draws out or shapes words carry into the generated speech. If you read a script, read it the way you would say it.

​

Speak with it

export ASSEMBLYAI_API_KEY=<your-key>
export ASSEMBLYAI_VOICE_ID=<your-voice-id>

import os

import assemblyai as aai

config = aai.TTSConfig(voice_id=os.environ["ASSEMBLYAI_VOICE_ID"])

with aai.TTSSession(config, api_key=os.environ["ASSEMBLYAI_API_KEY"]) as session:
    c = session.begin.configuration
    print(f"voice_id {c['voice_id']}, resolved {c['resolved_voice']['voice_id']}")
    print(f"language {c['language']}, model {c['model']}")
    print(f"voice expires at {c['voice_expires_at']}")

    with open("cloned.pcm", "wb") as out:
        for chunk in session.speak("Hi, thanks for calling. How can I help you today?"):
            out.write(chunk)  # raw 24 kHz 16-bit mono PCM

import { createWriteStream } from "node:fs";
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });
const session = client.tts.session({ voiceId: process.env.ASSEMBLYAI_VOICE_ID });

const { configuration: c } = await session.connect();
console.log(`voice_id ${c.voice_id}, resolved ${c.resolved_voice.voice_id}`);
console.log(`language ${c.language}, model ${c.model}`);
console.log(`voice expires at ${c.voice_expires_at}`);

const out = createWriteStream("cloned.pcm");
for await (const chunk of session.speak("Hi, thanks for calling. How can I help you today?")) {
  out.write(chunk); // raw 24 kHz 16-bit mono PCM
}
out.end();
await session.close();

curl -sS --fail-with-body -D headers.txt https://tts.us.assemblyai.com/v1/speech \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -d '{"text": "Hi, thanks for calling. How can I help you today?", "voice_id": "'"$ASSEMBLYAI_VOICE_ID"'", "container": "wav"}' \
  -o cloned.wav

import os

from openai import OpenAI

client = OpenAI(api_key=os.environ["ASSEMBLYAI_API_KEY"], base_url="https://tts.us.assemblyai.com/v1/openai")

with client.audio.speech.with_streaming_response.create(
    model="assemblyai-tts-1",
    voice={"id": os.environ["ASSEMBLYAI_VOICE_ID"]},
    input="Hi, thanks for calling. How can I help you today?",
    response_format="pcm",
) as response:
    print(f"voice_id {response.headers['x-tts-voice-id']}, language {response.headers['x-tts-language']}")
    response.stream_to_file("cloned.pcm")  # raw 24 kHz 16-bit mono PCM

voice_id 01a139bf-78a0-7d18-9a4e-6c0f2b8d1e37, resolved 01a139bf-78a0-7d18-9a4e-6c0f2b8d1e37
language english, model assemblyai-tts-1
voice expires at 1797154212

​

Language and model

​

Data zones

​

First audio

​

Consent period

​

Renew consent

# Continues the program above. You supply play().
renew = aai.TTSConfig(voice_id=os.environ["ASSEMBLYAI_VOICE_ID"], extend_biometric_consent=True)

with aai.TTSSession(renew) as session:  # a session
    for chunk in session.speak("Consent renewed."):
        play(chunk)
print(session.termination.voice_expires_at)  # the expiry after the renewal

response = aai.TextToSpeech().synthesize("Consent renewed.", renew)  # or an HTTP request
print(response.voice_expires_at)

// Continues the program above. You supply play().
const session = client.tts.session({ voiceId: process.env.ASSEMBLYAI_VOICE_ID, extendBiometricConsent: true }); // a session
await session.connect();
for await (const chunk of session.speak("Consent renewed.")) play(chunk);
const summary = await session.close();
console.log(summary?.voice_expires_at); // the expiry after the renewal

const response = await client.tts.synthesize({ // or an HTTP request
  text: "Consent renewed.",
  voice_id: process.env.ASSEMBLYAI_VOICE_ID,
  extend_biometric_consent: true,
});
console.log(response.voiceExpiresAt);

curl -sS --fail-with-body -D headers.txt https://tts.us.assemblyai.com/v1/speech \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -d '{"text": "Consent renewed.", "voice_id": "'"$ASSEMBLYAI_VOICE_ID"'", "extend_biometric_consent": true, "stream": false}' \
  -o renewed.pcm
grep -i x-tts-voice-expires-at headers.txt

​

Delete a voice

curl -sS --fail-with-body -X DELETE "https://tts.us.assemblyai.com/v1/voices/$ASSEMBLYAI_VOICE_ID" \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -o /dev/null -w '%{http_code}\n'

import os

import httpx  # ships with the Python SDK

response = httpx.delete(
    f"https://tts.us.assemblyai.com/v1/voices/{os.environ['ASSEMBLYAI_VOICE_ID']}",
    headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]},
)
response.raise_for_status()  # 204, no body
print("deleted")

const response = await fetch(`https://tts.us.assemblyai.com/v1/voices/${process.env.ASSEMBLYAI_VOICE_ID}`, {
  method: "DELETE",
  headers: { Authorization: process.env.ASSEMBLYAI_API_KEY },
});
if (response.status !== 204) throw new Error(`${response.status}: ${await response.text()}`);
console.log("deleted");

• New sessions and requests with the voice are refused within 6 minutes, in most cases at once, with voice_not_found, the same answer as for a voice that never existed. See Errors.
• Sessions already open keep speaking. The voice is checked when a session opens, so a session that has the voice finishes its conversation with it. Terminate the sessions yourself if they should stop.
• The slug is free again for a new voice of yours. The ID is never reused.
• Its data is removed. The sample recording and the voice model are deleted from storage afterwards, normally within two days.

​

Biometric data terms

This account has not accepted the biometric data terms. Please visit https://www.assemblyai.com/dashboard/settings/data-controls to review and accept them.

​

Errors

• voice_not_found (close 3012): no voice with this ID or slug that your account can use on this host. It does not exist, was deleted, is another account’s private voice, or belongs to the other data zone. Check the ID and the host.
• voice_expired (close 3012): the consent period ended. Create the voice again.
• voice_blocked (close 3012): the voice is blocked. Contact support.
• voice_not_paired (close 3014): the voice is not ready on the session’s model, usually because it was just created. Try again shortly; if it keeps failing, check the voice in the dashboard. The texts of this refusal are in Error handling.
• not_owner (close 3006): extend_biometric_consent was set on a voice another account owns. Leave it out.
• biometric_terms_not_accepted (close 3006, 403 over HTTP): your account has not accepted the biometric data terms. Accept them on the dashboard’s data controls page.
• invalid_parameter on voice_id (close 3006): the value is neither a voice ID nor a slug, or it is a slug with capital letters. Send the value the dashboard shows, a slug in lowercase.
• model_not_found, model_retired (close 3006, 3014): a model_id the region does not know, or a retired model. Leave model_id out.

​

Related

• Voices: the voice catalog and the AssemblyAI voices to start with.
• WebSocket API basics: how voice_id and extend_biometric_consent are checked at connect.
• Error handling: every close code and what to do about it.
• OpenAI compatibility: a cloned voice through the OpenAI SDKs.
• Cloud endpoints and data residency: the US and EU hosts.