Pipecat voice agent with AssemblyAI Universal-3.5 Pro Realtime

Build a real-time voice agent using Pipecat — the open-source Voice AI framework — and AssemblyAI's Universal-3.5 Pro Realtime model as the speech-to-text engine.

Pipecat's modular pipeline design means you can swap any component without touching the rest. AssemblyAI has a first-party Pipecat plugin with full Universal-3.5 Pro Realtime support — no manual WebSocket wiring required.

Why AssemblyAI in Pipecat?

Universal-3.5 Pro Realtime is AssemblyAI's flagship real-time model, purpose-built for voice agents. On Pipecat's own open STT benchmark of real agent conversations, it leads the field:

Beyond raw accuracy, it brings three things that matter for live conversation: punctuation-based turn detection (fewer awkward double-responses when users pause mid-thought), Context Carryover (the model hears each turn in the context of your agent's last question), and included keyterm prompting. Transcripts stream at roughly 150 ms P50.

Heads up on model IDs: the older u3-rt-pro model auto-routes to universal-3-5-pro on August 7, 2026 and stops accepting the old ID around September 25, 2026. This tutorial uses the current universal-3-5-pro string throughout.

Prerequisites

• Python 3.11+
• AssemblyAI API key (free account)
• OpenAI API key
• Cartesia API key

Quick start

git clone https://github.com/kelsey-aai/voice-agent-pipecat-universal-3-5-pro
cd voice-agent-pipecat-universal-3-5-pro

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt   # pins pipecat-ai>=1.4.0

cp .env.example .env
# Edit .env with your API keys

python bot.py

Speak into your microphone after the greeting. For WebRTC or Daily testing, see the integration guide.

Here's the core of the agent. Note the settings=AssemblyAISTTService.Settings(...) structure — this is the current plugin API (Pipecat 1.4.0+), and the assistant_aggregator at the end of the pipeline is what enables automatic conversation context:

import os
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.pipeline.pipeline import Pipeline
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.services.assemblyai.stt import AssemblyAISTTService
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.services.openai.llm import OpenAILLMService

stt = AssemblyAISTTService(
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        min_turn_silence=100,
    ),
    vad_force_turn_endpoint=True,   # Pipecat mode (default): VAD + Smart Turn
control turns
)

llm = OpenAILLMService(api_key=os.environ["OPENAI_API_KEY"])
tts = CartesiaTTSService(api_key=os.environ["CARTESIA_API_KEY"])

context = LLMContext()
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
)

pipeline = Pipeline([
    transport.input(),
    stt,
    user_aggregator,
    llm,
    tts,
    transport.output(),
    assistant_aggregator,   # → feeds each agent reply back as conversation
context automatically
])

Universal-3.5 Pro Realtime features

Keyterm prompting

Boost accuracy on domain-specific vocabulary — included at no extra cost, no restart required:

stt = AssemblyAISTTService(
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        keyterms_prompt=["AssemblyAI", "Universal-3.5 Pro", "Pipecat",
"YourBrandName"],
    ),
)

Essential for medical, legal, and financial applications where names and codes matter.

Conversation context (Context Carryover)

Universal-3.5 Pro Realtime keeps a short memory of the dialog and transcribes each user turn in the context of what your agent just said. After your agent asks *"What's your email address?"*, it can produce user@assemblyai.com instead of user at assemblyai dot com. Feeding the agent's question to the model cut WER by 10.2% on a 20,000-file voice-agent benchmark.

In Pipecat this is automatic — as long as your pipeline includes the assistant_aggregator from LLMContextAggregatorPair (as above), the plugin feeds each completed agent reply to the model as agent_context. To seed context for the very first user reply, set agent_context at construction time:

settings=AssemblyAISTTService.Settings(
    model="universal-3-5-pro",
    agent_context="Hi! Thanks for calling Acme. What's the email on your 
account?",
)

Voice Focus (noisy audio)

Server-side noise suppression that isolates the primary speaker before audio reaches the model — use it instead of client-side noise cancellation:

settings=AssemblyAISTTService.Settings(
    model="universal-3-5-pro",
    voice_focus="far-field",   # "near-field" for close-talking mics
)

Multilingual support

Universal-3.5 Pro Realtime code-switches natively across 18 languages — English, Spanish, French, German, Italian, Portuguese, and more — including mid-sentence switches. It's on by default; to bias toward a single language, pin language_code.

Speaker labels (optional)

For multi-party conversations, enable per-turn speaker labels:

settings=AssemblyAISTTService.Settings(
    model="universal-3-5-pro",
    speaker_labels=True,
)

Tuning turn detection

In Pipecat mode (vad_force_turn_endpoint=True, the default), Pipecat's VAD + Smart Turn analyzer decide when the user is done, and max_turn_silence is auto-synced to min_turn_silence. Start with the mode preset and fine-tune from there:

settings=AssemblyAISTTService.Settings(
    model="universal-3-5-pro",
    mode="balanced",       # "min_latency" · "balanced" · "max_accuracy"
    min_turn_silence=100,  # raise to 200–500 if entities split across turns
)

To let AssemblyAI's own punctuation-based turn detection control turn endings instead, set vad_force_turn_endpoint=False — then max_turn_silence is respected independently.

Note: end_of_turn_confidence_threshold and format_turns do not apply to Universal-3.5 Pro Realtime. Turn detection is punctuation-based and transcripts are always formatted; those parameters belong to the older universal-streaming models.

Deploy to Pipecat Cloud

pip install pipecatcloud
pcc auth login
pcc init
pcc secrets set my-agent-secrets --file .env
pcc deploy

Resources

• AssemblyAI Pipecat integration guide
• Pipecat docs
• Pipecat GitHub

Frequently asked questions

What is the best speech-to-text model for a Pipecat voice agent?

AssemblyAI's Universal-3.5 Pro Realtime (universal-3-5-pro) is purpose-built for voice agents and leads Pipecat's own open real-agent benchmark at 6.99% WER. It integrates through AssemblyAI's first-party Pipecat plugin (pipecat-ai 1.4.0+) with no manual WebSocket wiring, streams partial and final transcripts around 150 ms P50, and adds Context Carryover and included keyterm prompting — the features that most affect whether a voice agent understands the user.

How do I add AssemblyAI to a Pipecat pipeline?

Install the plugin (pip install "pipecat-ai[assemblyai,openai,cartesia]"), then add AssemblyAISTTService to your pipeline with settings=AssemblyAISTTService.Settings(model="universal-3-5-pro"). Include the LLMContextAggregatorPair assistant aggregator at the end of the pipeline to enable automatic conversation context. That's the whole integration — the plugin handles the streaming connection for you.

Does AssemblyAI's Pipecat plugin support conversation context?

Yes, and it's automatic. As long as your pipeline includes the standard LLM context aggregator, the plugin feeds each completed agent reply to Universal-3.5 Pro Realtime as agent_context, so the model transcribes the next user turn with the agent's question in context. This is especially impactful on short answers and spelled-out entities like emails and account numbers.

How does Universal-3.5 Pro Realtime handle turn detection in Pipecat?

You choose which component decides turns with vad_force_turn_endpoint. In the default Pipecat mode (True), Pipecat's VAD + Smart Turn analyzer control turns. Set it to False to use AssemblyAI's built-in punctuation-based turn detection, where the model checks for terminal punctuation after a short silence and respects min_turn_silence / max_turn_silence as configured.

How much does an AssemblyAI Pipecat voice agent cost?

Universal-3.5 Pro Realtime is $0.45/hr, billed on session duration with no minimums and unlimited concurrency; keyterm prompting is included. Your LLM and TTS providers (e.g. OpenAI and Cartesia) bill separately. AssemblyAI has a free tier so you can build and test before adding a card.

‍