Best Practices for Building Voice Agents

Introduction

AssemblyAI’s Universal-3 Pro Streaming is the most accurate real-time speech-to-text model designed for voice agents. It delivers formatted, immutable transcripts with sub-300ms latency, exceptional entity accuracy, native multilingual code switching, and a fully promptable interface, all optimized for conversational AI workflows.

The STT component is the “ears” of your voice agent. Transcription errors propagate into the LLM and response logic, so even small accuracy gaps compound in impact. Choosing and configuring the right STT model is one of the highest-leverage decisions you can make when building a voice agent. For guidance on how to evaluate and compare STT models for your use case, see the streaming evaluation guide.

Why Universal-3 Pro Streaming for Voice Agents?

Voice agents need speed, accuracy, and natural turn-taking. Universal-3 Pro Streaming is purpose-built for this:

Sub-300ms latency with formatted output

• Immutable transcripts arrive fully formatted (punctuation, capitalization), no waiting for a separate formatting step
• Every final transcript is ready for immediate LLM processing

Exceptional entity accuracy

• Credit card numbers, phone numbers, email addresses, physical addresses, and names are transcribed with high accuracy
• Short utterances like “yes”, “no”, “mmhmm” are handled reliably

Punctuation-based turn detection

• Turn boundaries are determined by terminal punctuation (. ? !) combined with silence thresholds
• Configurable min_turn_silence and max_turn_silence parameters let you tune responsiveness vs. accuracy
• No confidence-score guessing. The model understands when a sentence is complete

Fully promptable

• Custom prompt parameter for transcription instructions
• Dynamic prompting mid-session via UpdateConfiguration. Adapt the model to each stage of the conversation
• keyterms_prompt for boosting recognition of specific names, brands, and domain terms

Native multilingual support

• Supports English, Spanish, French, German, Italian, and Portuguese
• Automatic code-switching between languages within a single session
• Language-specific prompting for improved accuracy

What Languages Does Universal-3 Pro Streaming Support?

Universal-3 Pro Streaming supports six languages with automatic code-switching:

• English
• Spanish
• French
• German
• Italian
• Portuguese

The model handles code-switching natively. Speakers can switch between supported languages mid-conversation without any configuration changes. Accuracy improves when you specify the expected language in the prompt. See Supported languages for the full language list and regional dialect reference.

To guide the model toward a specific language, prepend language information to the default prompt:

Transcribe Spanish. Transcribe verbatim with standard punctuation. Include filler words and incomplete utterances.

For multilingual conversations:

Transcribe multilingual conversation in Spanish and English. Transcribe verbatim with standard punctuation. Include filler words and incomplete utterances.

How Do I Get Started?

Complete voice agent stack

AssemblyAI provides a speech-to-speech Voice Agent API that abstracts away the complexity of a full voice agent stack: managed STT, LLM, turn detection, and TTS in a single endpoint.

For a cascading architecture, AssemblyAI has the best speech-to-text model. For a complete stack, you need:

1. Speech-to-Text (STT): AssemblyAI Universal-3 Pro Streaming
2. Large Language Model (LLM): OpenAI, Anthropic, Google, etc.
3. Text-to-Speech (TTS): Rime, Cartesia, ElevenLabs, etc.
4. Orchestration: LiveKit, Pipecat, or custom build

Pre-built integrations

LiveKit Agents (recommended)

LiveKit provides the fastest path to a working voice agent with AssemblyAI. See Universal-3 Pro Streaming on LiveKit for a full guide.

1
from livekit.agents import AgentSession
2
from livekit.plugins import assemblyai, silero
3
4
session = AgentSession(
5
    stt=assemblyai.STT(
6
        model="u3-rt-pro",
7
        min_turn_silence=100,
8
        max_turn_silence=1000,
9
        vad_threshold=0.3,
10
    ),
11
    vad=silero.VAD.load(
12
        activation_threshold=0.3,
13
    ),
14
    turn_detection="stt",
15
    min_endpointing_delay=0,
16
)

Pipecat by Daily

Pipecat is an open-source framework for conversational AI with maximum customizability. See Universal-3 Pro Streaming on Pipecat for a full guide.

1
from pipecat.services.assemblyai.stt import AssemblyAISTTService
2
from pipecat.services.assemblyai.models import AssemblyAIConnectionParams
3
4
stt = AssemblyAISTTService(
5
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
6
    connection_params=AssemblyAIConnectionParams(
7
        speech_model="u3-rt-pro",
8
        min_turn_silence=100,
9
        max_turn_silence=1000,
10
    ),
11
    vad_force_turn_endpoint=False,  # Use AssemblyAI's built-in turn detection
12
)

Direct WebSocket connection

For custom builds, connect directly to the WebSocket API:

1
import json
2
import pyaudio
3
import websocket
4
import threading
5
import time
6
from urllib.parse import urlencode
7
8
API_KEY = "YOUR_API_KEY"
9
SAMPLE_RATE = 16000
10
11
CONNECTION_PARAMS = {
12
    "sample_rate": SAMPLE_RATE,
13
    "speech_model": "u3-rt-pro",
14
    "min_turn_silence": 100,
15
    "max_turn_silence": 1000,
16
}
17
18
API_ENDPOINT_BASE = "wss://streaming.assemblyai.com/v3/ws"
19
API_ENDPOINT = f"{API_ENDPOINT_BASE}?{urlencode(CONNECTION_PARAMS)}"
20
21
def on_message(ws, message):
22
    data = json.loads(message)
23
24
    if data.get("type") == "Turn":
25
        transcript = data.get("transcript", "")
26
        end_of_turn = data.get("end_of_turn", False)
27
28
        if end_of_turn:
29
            # Final transcript - send to LLM
30
            print(f"Final: {transcript}")
31
        else:
32
            # Partial - can start pre-emptive LLM generation
33
            print(f"Partial: {transcript}")
34
35
    elif data.get("type") == "SpeechStarted":
36
        # User started speaking - handle barge-in
37
        print("Speech detected, interrupt agent if speaking")
38
39
ws = websocket.WebSocketApp(
40
    API_ENDPOINT,
41
    header={"Authorization": API_KEY},
42
    on_message=on_message,
43
)

How Does Turn Detection Work?

Universal-3 Pro Streaming uses a punctuation-based turn detection system controlled by two parameters:

How it works:

1. User speaks → audio streams to AssemblyAI
2. User pauses for min_turn_silence → model checks for terminal punctuation (. ? !)
3. If terminal punctuation found → turn ends immediately with end_of_turn: true
4. If no terminal punctuation → partial emitted with end_of_turn: false, turn continues
5. If silence reaches max_turn_silence → turn forced to end regardless of punctuation

Configuration presets

1
# Fast - quick confirmations, IVR, yes/no questions
2
fast_params = {
3
    "speech_model": "u3-rt-pro",
4
    "min_turn_silence": 100,
5
    "max_turn_silence": 800,
6
}
7
8
# Balanced - most voice agent conversations (recommended)
9
balanced_params = {
10
    "speech_model": "u3-rt-pro",
11
    "min_turn_silence": 100,
12
    "max_turn_silence": 1000,
13
}
14
15
# Patient - entity dictation, complex instructions
16
patient_params = {
17
    "speech_model": "u3-rt-pro",
18
    "min_turn_silence": 200,
19
    "max_turn_silence": 2000,
20
}

Entity splitting tradeoff

Lower silence values produce faster transcripts but can split entities across turns:

# With (min_turn_silence=100, max_turn_silence=1000)
"It's John."                    → turn ends (period found after 100ms pause)
"Smith."                        → new turn
"At gmail.com."                 → new turn
# With (min_turn_silence=400, max_turn_silence=2000)
"It's john.smith@gmail.com."    → single turn (properly formatted)

For voice agents, the downstream LLM can usually piece together split entities. But if your use case involves entity extraction or alphanumeric dictation, increase min_turn_silence and max_turn_silence during those portions of the conversation using dynamic configuration updates.

How Do I Handle Barge-In and Interruptions?

SpeechStarted events

Universal-3 Pro Streaming emits SpeechStarted events when speech has been detected. SpeechStarted is only emitted when the model produces a transcript. This makes it a reliable signal for barge-in handling:

1
{
2
  "type": "SpeechStarted",
3
  "timestamp": 14400,
4
  "confidence": 0.79
5
}

When you receive a SpeechStarted event:

1. Stop TTS playback immediately
2. Switch the agent back to listening mode
3. Wait for the user’s full turn to complete before responding

Filtering backchannels

Users often produce short backchannel utterances (“mhm”, “yeah”, “um”, “okay”) while the agent is speaking. Treating every SpeechStarted event as a barge-in causes the agent to stop mid-sentence on these fillers, even though the user didn’t intend to interrupt.

The fix is to gate barge-in on each Turn event during agent speech: suppress the interrupt when the transcript is short or every token is a known backchannel. Implementation depends on your stack:

• Voice Agent API: semantic interruption classification is built in. See Turn detection and interruptions.
• LiveKit: LiveKit Cloud users should enable adaptive interruption handling. Self-hosted deployments use the two custom filters in the same section.
• Direct WebSocket: see Interruption handling on the Universal-3 Pro Streaming API page for the combined word-count + filler-word filter.

VAD threshold alignment

Universal-3 Pro Streaming includes an internal Silero VAD controlled by the vad_threshold parameter (default 0.3). If you’re also running a local VAD (common in LiveKit and Pipecat), align the thresholds to avoid a dead zone where one detects speech but the other doesn’t:

1
# Both thresholds aligned at 0.3
2
stt = assemblyai.STT(
3
    model="u3-rt-pro",
4
    vad_threshold=0.3,
5
)
6
vad = silero.VAD.load(
7
    activation_threshold=0.3,
8
)

If you’re in a noisy environment and getting false speech triggers, raise both thresholds together.

How Can I Use Prompting to Improve Accuracy?

The prompt parameter

Universal-3 Pro Streaming supports a prompt parameter for custom transcription instructions. When no prompt is provided, a default prompt optimized for turn detection is applied automatically.

1
CONNECTION_PARAMS = {
2
    "speech_model": "u3-rt-pro",
3
    "prompt": "Transcribe this audio: AI voice agent talking to a human for customer service. Mandatory: Transcribe verbatim with all spoken filler words, hesitations, repetitions, and false starts exactly as spoken."
4
}

Tips for effective prompts:

• Specify the audio context: accent, domain, expected utterance types
• Define punctuation rules: improves downstream LLM processing
• Preserve speech patterns: instruct the model to keep filler words for more natural interactions
• Specify language: prepend Transcribe <language>. for non-English or multilingual conversations

Keyterms prompting

Use keyterms_prompt to boost recognition of specific names, brands, or domain terms, up to 100 terms per session:

1
CONNECTION_PARAMS = {
2
    "speech_model": "u3-rt-pro",
3
    "keyterms_prompt": json.dumps([
4
        "AssemblyAI",
5
        "LiveKit",
6
        "Dr. Rodriguez",
7
        "Lisinopril",
8
        "iPhone 15 Pro",
9
    ])
10
}

Best practices for keyterms:

• Include proper names, product names, technical terms, and domain-specific jargon
• Include terms up to 50 characters each
• Don’t include common English words, single letters, or generic phrases
• Don’t exceed 100 terms total

For detailed guidance, see Keyterms prompting.

How Do I Update Configuration Mid-Session?

You can update prompt, keyterms_prompt, min_turn_silence, and max_turn_silence during an active session using UpdateConfiguration. This is one of Universal-3 Pro Streaming’s most powerful features for voice agents.

Dynamic keyterms by conversation stage

As your voice agent moves through different stages, update keyterms to match what the user is likely to say:

1
# Caller identification stage
2
ws.send(json.dumps({
3
    "type": "UpdateConfiguration",
4
    "keyterms_prompt": ["Kelly Byrne-Donoghue", "date of birth", "January", "February"]
5
}))
6
7
# Medical intake stage
8
ws.send(json.dumps({
9
    "type": "UpdateConfiguration",
10
    "keyterms_prompt": ["cardiology", "echocardiogram", "Dr. Patel", "metoprolol"]
11
}))
12
13
# Payment stage - also increase max_turn_silence for credit card dictation
14
ws.send(json.dumps({
15
    "type": "UpdateConfiguration",
16
    "keyterms_prompt": ["Visa", "Mastercard", "American Express"],
17
    "max_turn_silence": 3000
18
}))

Dynamic prompting

You can also update the transcription prompt mid-session. This is especially powerful when paired with tool calls in your LLM:

• If your agent asks a yes/no question, prompt the model to anticipate short responses
• If your agent asks for a phone number or email, prompt it to expect those formats
• If you present a list of options, boost those options in the prompt

1
# After asking "Would you like to confirm your appointment?"
2
ws.send(json.dumps({
3
    "type": "UpdateConfiguration",
4
    "prompt": "User is responding yes or no to a confirmation question. Expect short responses."
5
}))
6
7
# After asking "What's your phone number?"
8
ws.send(json.dumps({
9
    "type": "UpdateConfiguration",
10
    "prompt": "User is dictating a phone number. Expect digits and formatting.",
11
    "max_turn_silence": 3000
12
}))

How Do I Use Speaker Diarization?

Streaming Diarization identifies and labels individual speakers in real time. Each Turn event includes a speaker_label field (e.g., "A", "B") indicating which speaker produced that transcript.

Enable it by adding speaker_labels: true to your connection parameters:

1
CONNECTION_PARAMS = {
2
    "speech_model": "u3-rt-pro",
3
    "speaker_labels": True,
4
}

Speaker accuracy improves over the course of a session as the model accumulates embedding context.

With LiveKit:

1
stt = assemblyai.STT(
2
    model="u3-rt-pro",
3
    speaker_labels=True,
4
)

With Pipecat (including custom formatting):

1
stt = AssemblyAISTTService(
2
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
3
    connection_params=AssemblyAIConnectionParams(
4
        speech_model="u3-rt-pro",
5
        speaker_labels=True,
6
    ),
7
    speaker_format="[{speaker}] {text}",
8
)

For more details, see Streaming Diarization and Multichannel.

How Do I Optimize for Latency?

Key optimizations

1. Use the right silence thresholds

Start with min_turn_silence=100 and max_turn_silence=1000. Only increase if you’re seeing entity splitting issues.

2. Tune interruption_delay for faster TTFT

The interruption_delay parameter controls how soon the first partial is emitted. Set interruption_delay=0 for the fastest possible time to first token (~300ms effective). The default of 500ms produces a first partial at ~800ms. See Tuning early partial timing for details.

3. Eliminate additive delays in your orchestrator

In LiveKit with turn_detection="stt", set min_endpointing_delay=0. LiveKit’s default 0.5s delay is additive on top of AssemblyAI’s own endpointing.

4. Use 16kHz sample rate

This balances audio quality and bandwidth. Higher sample rates don’t improve accuracy.

5. Align VAD thresholds

Mismatched VAD thresholds between your local VAD and AssemblyAI create a dead zone that delays interruption. Set both to 0.3.

6. Skip unnecessary features

Only enable speaker_labels if you need diarization. Only use keyterms_prompt if you have domain-specific terms. Each feature adds marginal processing overhead.

Latency breakdown

How Does the Message Sequence Work?

Universal-3 Pro Streaming sends messages in a specific sequence. Here’s what a typical conversation looks like:

1. Session begins

1
{
2
  "type": "Begin",
3
  "id": "session-id",
4
  "expires_at": 1759796682
5
}

2. Speech detected

1
{
2
  "type": "SpeechStarted",
3
  "timestamp": 1200,
4
  "confidence": 0.85
5
}

3. Early partial (emitted after 750ms of continuous speech)

1
{
2
  "type": "Turn",
3
  "turn_order": 0,
4
  "end_of_turn": false,
5
  "turn_is_formatted": false,
6
  "transcript": "Yeah my credit card..."
7
}

4. Silence-based partial (speaker pauses, no terminal punctuation)

1
{
2
  "type": "Turn",
3
  "turn_order": 0,
4
  "end_of_turn": false,
5
  "turn_is_formatted": false,
6
  "transcript": "Yeah my credit card number is--"
7
}

5. Final transcript (terminal punctuation found, or max_turn_silence reached)

1
{
2
  "type": "Turn",
3
  "turn_order": 0,
4
  "end_of_turn": true,
5
  "turn_is_formatted": true,
6
  "transcript": "Yeah, my credit card number is 8888-8888-8888-8888.",
7
  "speaker_label": "A"
8
}

6. Session termination

1
{
2
  "type": "Termination",
3
  "audio_duration_seconds": 45.2
4
}

For the complete message reference, see Message sequence.

How Can I Improve Accuracy?

Keyterms prompting

The single most effective way to improve accuracy on domain-specific terms. Keyterms are especially useful for improving recognition of proper nouns, product names, and technical jargon spoken with accents or in noisy environments. See How Can I Use Prompting to Improve Accuracy? above.

Dynamic configuration updates

Update keyterms and prompts mid-session based on conversation context. See How Do I Update Configuration Mid-Session? above.

Tune silence thresholds

If entities are splitting across turns, increase min_turn_silence (for punctuation-triggered splits) or max_turn_silence (for forced timeout splits). You can do this dynamically mid-session for specific conversation stages like entity dictation.

Noise handling

Universal-3 Pro Streaming handles background noise well out of the box. Avoid adding noise cancellation as a preprocessing step. The artifacts it introduces typically cause more harm than the background noise itself.

For telephony environments with low-quality audio (such as 8 kHz mulaw), you can prompt the model to tag genuinely unclear segments as [unclear] rather than forcing a guess. This helps you identify audio segments that no model (or human) can reliably transcribe, and prevents inaccurate guesses from entering your downstream pipeline.

Scaling and Concurrency

Universal-3 Pro Streaming provides unlimited concurrent streams:

• No hard caps on simultaneous connections
• No overage fees for spike traffic
• Automatic scaling from 5 to 50,000+ streams

Rate limits:

• Free users: 5 new streams per minute
• Pay-as-you-go: 100 new streams per minute
• When using 70%+ of your limit, capacity automatically increases 10% every 60 seconds

These limits are designed to never interfere with legitimate applications. Your baseline limit is guaranteed and never decreases, so you can scale smoothly without artificial barriers.

Evaluating your voice agent

Benchmark scores are a useful starting point, but they don’t tell the full story. To determine which STT model works best for your voice agent in production:

• Run A/B tests at scale. Swap models in your agent pipeline, compare core outcome metrics (task completion rate, booking rate, resolution rate), and let real user behavior determine the winner.
• Optimize for outcomes, not benchmarks. The question is not which model has the best WER. It is which model drives users to complete their goal most reliably.
• Simulate real scenarios early. Before you have real users, manually test the full agent flow under realistic conditions. This surfaces issues that isolated STT benchmarks will miss.

For a complete evaluation framework including accuracy metrics, latency metrics, and ground truth best practices, see the streaming evaluation guide.

Additional Resources

• Streaming evaluation guide
• Universal-3 Pro Streaming overview
• Turn detection guide
• Prompting guide
• Keyterms prompting
• Message sequence
• Streaming diarization and multichannel
• Supported languages
• LiveKit integration
• Pipecat integration
• API Playground
• Changelog