Universal-3 Pro Streaming on Pipecat

Pipecat

View Pipecat’s AssemblyAI STT plugin documentation.

Overview

This guide covers integrating AssemblyAI’s Universal-3 Pro Streaming (u3-rt-pro) speech-to-text model into Pipecat voice agents.

Universal-3 Pro Streaming is optimized for real-time audio utterances typically under 10 seconds, with special efficiencies built in for low-latency turn detection and voice agent workflows. It provides the highest accuracy with native multilingual code switching, entity accuracy, and prompting support.

Universal-3 Pro Streaming delivers exceptional entity and alphanumeric accuracy, including credit card numbers, cell phone numbers, email addresses, physical addresses, and names. All with sub-300ms time to complete transcript latency.

Key features

• Two-mode turn detection: Choose between Pipecat-controlled (VAD + Smart Turn) or AssemblyAI’s built-in turn detection (STT-based)
• Keyterms boosting: Improve recognition of specific words and names
• Dynamic parameter updates: Change configuration mid-conversation without reconnection
• Speaker diarization: Identify and label different speakers in multi-party conversations

Examples

Complete working examples are available in the Pipecat repository:

• voice-assemblyai.py: Voice agent with U3-Pro using Pipecat-based turn detection (VAD + Smart Turn)
• voice-assemblyai-turn-detection.py: Voice agent with U3-Pro using AssemblyAI’s built-in turn detection (STT-based)

You can run any example directly as long as your API keys are saved in a .env file:

$
python voice-assemblyai.py

Installation

Install Pipecat with all required dependencies:

$
pip install "pipecat-ai[assemblyai,openai,cartesia]"

What’s included:

• assemblyai: AssemblyAI U3-Pro STT service
• openai: OpenAI LLM service (used in the examples)
• cartesia: Cartesia TTS service (used in the examples)

Authentication

Set your API keys in a .env file:

1
ASSEMBLYAI_API_KEY=your_assemblyai_key
2
OPENAI_API_KEY=your_openai_key
3
CARTESIA_API_KEY=your_cartesia_key

Two-mode turn detection

Within Pipecat, you have two distinct approaches to turn detection with AssemblyAI’s U3-Pro model.

Pipecat mode (default, recommended)

When to use: Most voice agent applications requiring responsive interruptions.

1
stt = AssemblyAISTTService(
2
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
3
    settings=AssemblyAISTTService.Settings(
4
        model="u3-rt-pro",
5
        min_turn_silence=100,
6
    ),
7
    vad_force_turn_endpoint=True,  # Default (Pipecat mode)
8
)

How it works:

• VAD + Smart Turn analyzer controls when the user is done speaking
• ForceEndpoint message sent to AssemblyAI on VAD silence detection
• max_turn_silence automatically synchronized with min_turn_silence
• Best for low-latency, responsive voice agents

AssemblyAI’s built-in turn detection (STT mode)

When to use: When you want AssemblyAI’s built-in turn detection to control turn endings. This mode is configurable within the settings. See Configuring turn detection to understand how it works.

1
stt = AssemblyAISTTService(
2
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
3
    settings=AssemblyAISTTService.Settings(
4
        model="u3-rt-pro",
5
        min_turn_silence=100,
6
        max_turn_silence=1000,  # Now respected
7
    ),
8
    vad_force_turn_endpoint=False,  # AssemblyAI's built-in turn detection (STT mode)
9
)

How it works:

• AssemblyAI’s built-in turn detection controls when the user is done speaking
• All timing parameters are respected as configured
• Emits UserStartedSpeakingFrame / UserStoppedSpeakingFrame
• Uses SpeechStarted events for fast barge-in
• Only available with u3-rt-pro (other models require Pipecat mode)

Keyterms boosting

Improve recognition of specific words or names:

1
stt = AssemblyAISTTService(
2
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
3
    settings=AssemblyAISTTService.Settings(
4
        model="u3-rt-pro",
5
        min_turn_silence=100,
6
        keyterms_prompt=["Xiomara", "Saoirse", "Pipecat", "AssemblyAI"],
7
    ),
8
)

Dynamic parameter updates

Change configuration mid-conversation without reconnection. See stt-assemblyai.py for a complete working example.

1
from pipecat.frames.frames import STTUpdateSettingsFrame
2
from pipecat.services.assemblyai.stt import AssemblyAISTTService
3
4
# Update keyterms during conversation
5
await task.queue_frame(
6
    STTUpdateSettingsFrame(
7
        delta=AssemblyAISTTService.Settings(
8
            keyterms_prompt=["NewName", "NewCompany"]
9
        )
10
    )
11
)
12
13
# Update silence thresholds
14
await task.queue_frame(
15
    STTUpdateSettingsFrame(
16
        delta=AssemblyAISTTService.Settings(
17
            min_turn_silence=200,
18
            max_turn_silence=3000,  # Only respected in AssemblyAI's built-in turn detection (STT mode)
19
        )
20
    )
21
)

Speaker diarization

Identify different speakers in multi-party conversations.

Basic diarization

1
stt = AssemblyAISTTService(
2
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
3
    settings=AssemblyAISTTService.Settings(
4
        model="u3-rt-pro",
5
        speaker_labels=True,
6
    ),
7
)

Speaker labels (e.g., "A", "B", "C") are included in final transcripts and logged.

With custom formatting

Format transcripts with speaker labels for LLM context:

1
stt = AssemblyAISTTService(
2
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
3
    settings=AssemblyAISTTService.Settings(
4
        model="u3-rt-pro",
5
        speaker_labels=True,
6
    ),
7
    speaker_format="<{speaker}>{text}</{speaker}>",
8
)

Format options:

Daily transport

For production deployments, use the Daily transport for WebRTC-based real-time audio/video communication.

Parameters reference

U3-Pro specific parameters

General parameters

Running your agent

Development mode (local audio)

$
python your_agent.py

Speak into your microphone after hearing the greeting.

Production with Daily

Deploy to Daily.co rooms using the Daily transport. Your agent joins as a participant and handles audio I/O through Daily’s infrastructure.

Speech model comparison

Interested in using a different model?

Legend:

• ✅ Fully supported and recommended
• ❌ Not supported / Not used