Optimizing Accuracy and Latency

​

Measure the time to first audio

• HTTP API. Every 200 carries a Server-Timing header. On a streamed response, its ttfa is our server’s own time from receiving the request to sending the first audio byte. Your client’s time to the first byte, minus ttfa, is the network, the connection setup and your client. When that difference is large, the connection and data zone habits below are the ones that help. See Response headers.
• WebSocket API. Time a turn from the Generate you send to the first Audio frame, and time opening a session, from connecting to Begin, separately. The first is what a turn waits for on an open session. The second is what a turn adds when it opens its own. When the text streams in from an LLM, also time the first audio from the moment the first sentence is complete: that is the wait flushing each sentence shortens.

import os
import time

import httpx

client = httpx.Client(headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]})
body = {"text": "Hi, thanks for calling. How can I help you today?", "voice_id": "aai-audrey"}

for _ in range(5):
    start = time.perf_counter()
    with client.stream("POST", "https://tts.us.assemblyai.com/v1/speech", json=body) as response:
        next(response.iter_bytes())  # the first audio bytes
        client_ms = (time.perf_counter() - start) * 1000
        server_timing = response.headers["server-timing"]  # "open;dur=2.7, model;dur=40.4, ttfa;dur=43.1"
    print(f"client {client_ms:.0f} ms  server {server_timing}")

​

Use one WebSocket session per conversation

​

Open the session before the first reply

# Fragment. You supply call_is_open(), next_reply() and play().
import os

import assemblyai as aai

config = aai.TTSConfig(voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", language="english")  # Audrey

# When the call starts, not when the first reply is ready.
session = aai.TTSSession(config, api_key=os.environ["ASSEMBLYAI_API_KEY"])
session.connect()  # waits for Begin; the server then loads the voice before the first reply

while call_is_open():
    reply = next_reply(timeout=60)  # your agent; None after 60 s of silence
    if reply is None:
        session.keep_alive()  # restarts the idle timer, nothing else
        continue
    for chunk in session.speak(reply):
        play(chunk)

session.terminate()

// Fragment. You supply callIsOpen(), nextReply() and play().
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });

// When the call starts, not when the first reply is ready.
const session = client.tts.session({ voiceId: "49c443a4-8a1d-5f7e-b70b-98bdbd9823c3", language: "english" }); // Audrey
await session.connect(); // waits for Begin; the server then loads the voice before the first reply

while (callIsOpen()) {
  const reply = await nextReply(60_000); // your agent; null after 60 s of silence
  if (reply === null) {
    session.keepAlive(); // restarts the idle timer, nothing else
    continue;
  }
  for await (const chunk of session.speak(reply)) play(chunk);
}

await session.close();

​

Reuse one TLS context in Python

import ssl

import websockets

# Once per process. One context can be shared by any number of connections.
SSL_CONTEXT = ssl.create_default_context()


async def open_session(url, headers):
    """Open a session; the caller reads Begin and closes the connection."""
    return await websockets.connect(url, additional_headers=headers, ssl=SSL_CONTEXT)

​

Keep HTTP connections open

import httpx

# One client per process: httpx's own pool sizes, with idle connections kept
# for 2 minutes instead of 5 seconds.
client = httpx.Client(
    limits=httpx.Limits(max_connections=100, max_keepalive_connections=20, keepalive_expiry=120),
    timeout=httpx.Timeout(5.0, read=30.0),
)

import { Agent, fetch } from "undici";

// One agent per process: idle connections kept for 2 minutes instead of 4 seconds.
// Pass undici's own fetch with its Agent; npm install undici.
const agent = new Agent({ keepAliveTimeout: 120_000 });

const response = await fetch("https://tts.us.assemblyai.com/v1/speech", {
  method: "POST",
  dispatcher: agent,
  headers: { Authorization: process.env.ASSEMBLYAI_API_KEY, "Content-Type": "application/json" },
  body: JSON.stringify({ text: "Hello from AssemblyAI." }),
});

client.head("https://tts.us.assemblyai.com/v1/openai/models")

​

Send text as it is written, and flush each sentence

​

Stream raw PCM in the format your player needs

​

Use the nearest data zone

​

In a browser

• Keep a spare token. Mint it on an early hint that a session will be needed, such as the pointer entering the text box, and only when the page will have to open a session, such as after the idle session closed. While a session is open, a spare goes unused and only sits in the page as a live credential. Drop it well before its expires_in_seconds runs out, and give every connection a token of its own.
• Reopen on intent. A page that lets an idle session close and reopens it when the user presses Speak makes that click wait for the token, the handshakes and Begin. Reopen when the user focuses the text box instead, while they type, and the click finds the session open. A click that comes before the reopened session’s Begin should keep its text and speak it when Begin arrives.

​

Write the text the way it should be read

• Read in their spoken form: amounts with a currency sign, dates written with their year, phone numbers, addresses, units, email addresses and URLs. $25.50 is read as twenty five dollars and fifty cents, and (415) 555-0132 in groups of digits. An English numeric date such as 03/04/2026 is read month first, as March fourth.
• Also read out in English: common written shorthand and technical strings. p. 12 is read as page 12, w/ as with and press * as press star, and a file path, a MAC address or a 0x hex number symbol by symbol.
• Dropped: emoji, Markdown and HTML markup, footnote marks such as the [3] in as the study showed[3], and decorative symbols.
• Left to the model: anything else, such as a number with no format that says what it is, like 4321 or 1,234. The model reads it on its own.

• Use full sentences with punctuation. The model takes pace and tone from it. Text with no punctuation may not be read in the sentences you expect, and a question mark, an exclamation mark or a comma each change the delivery.
• Spell out a reading the model could get wrong. Put dashes between digits to read a number digit by digit, as in a card number, and write it with commas to read it as a whole number. Put dashes between the letters of an uncommon acronym. Write out a date or an amount in words when the written form is ambiguous, such as a numeric date meant day first or one written without its year.
• Remove anything that should not be spoken. Stage directions are read aloud: square brackets become parentheses, so [pause] is read as the word pause, not as a pause instruction.
• Send complete sentences per request. A request that ends mid-sentence is spoken on its own, without the context of the sentence it belongs to, which is the same reason not to flush per token.

​

Set the language of the text

​

Related

• Message frames: when the server starts speaking a Generate, and what Flush, Cancel and KeepAlive do.
• Limits: text, session and rate limits, and how each one reports a breach.
• Prompting guide: pacing, tone and normalization.
• WebSocket API basics: the encodings and sample rates, and playing raw PCM from Begin’s sample rate.
• OpenAI compatibility: keeping connections with the OpenAI SDKs, Pipecat and LiveKit.
• HTTP API: streamed and buffered responses, audio formats and Server-Timing.