LiveKit voice agent with AssemblyAI Universal-3 Pro Realtime

LiveKit is how a huge share of the voice-agent market gets to production — and AssemblyAI's Universal-3.5 Pro Realtime is a native streaming speech-to-text plugin in LiveKit Agents. This guide shows you how to wire the two together, tune the settings that actually matter, and avoid the one integration gotcha that trips most people up.

Why this pairing? Most teams try AssemblyAI through a framework like LiveKit — if that path is smooth, they stay; if it breaks, they revert. So it's worth getting right. LiveKit's own team put it this way, on making Universal-3.5 Pro available on LiveKit Inference: "What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms." — David Zhao, Co-founder at LiveKit.

The model and endpoint

• Model: Universal-3.5 Pro Realtime, id universal-3-5-pro (the streaming parameter is speech_model, singular).
• Endpoint: wss://streaming.assemblyai.com/v3/ws. The old /v2/realtime/ws returns HTTP 410 — don't use it.
• Audio: PCM16 mono 16kHz; auth is your raw API key (no Bearer prefix on streaming STT).
• Pricing: $0.45/hr base for streaming, or the managed Voice Agent API at a flat $4.50/hr if you'd rather not assemble STT + LLM + TTS yourself.

Streaming setup in Python

The core streaming loop looks like this — it's the same speech layer the LiveKit plugin runs under the hood:

# pip install "assemblyai>=1.0.0"
import os
from assemblyai.streaming.v3 import (
    StreamingClient, StreamingClientOptions, StreamingEvents,
    StreamingParameters, TurnEvent,
)

def on_turn(_, event: TurnEvent):
    tag = "FINAL" if event.end_of_turn else "partial"
    print(f"{tag}: {event.transcript}")

client = StreamingClient(StreamingClientOptions(api_key=os.environ["ASSEMBLYAI_API_KEY"]))
client.on(StreamingEvents.Turn, on_turn)
client.connect(StreamingParameters(
    sample_rate=16000,
    speech_model="universal-3-5-pro",
    agent_context="What's your email address?",
))
# feed 16kHz mono PCM16 chunks (50-1000ms) via client.stream(chunk)
# after each agent reply, update the context:
client.update_configuration(agent_context="Sure, what date would you like to book?")
client.disconnect(terminate=True)  # ALWAYS terminate

agent_context / Context Carryover

The capability David Zhao called out is worth understanding. Universal-3.5 Pro Realtime takes an agent_context parameter — you feed it what your agent just said and it biases recognition toward the expected reply. Across 20,000 voice-agent files, this cut WER by 10.2%. Rolling context is on by default (prior finalized user turns carry over automatically), and LiveKit supports it out of the box, so you get it without predefining key terms.

The turn-detection gotcha (read this)

Do not use turn_detection="stt" with the Pro model. Use LiveKit's English turn-detection model instead. This is the single most common way a LiveKit + AssemblyAI setup goes wrong, so wire it up correctly from the start.

How accurate is it?

On the open Pipecat voice-agent benchmark, Universal-3.5 Pro Realtime posts 6.99% WER versus Deepgram Flux at 15.58%, and it's the only model in Coval's independent Human Parity Zone (3.40% WER, ~110ms p50). Note that Pipecat and Coval measured Universal-3.5 Pro Realtime at roughly 425–450ms time-to-final — slightly slower than the previous generation but more accurate. Full numbers are in our independent STT benchmarks.

Diarization and deployment

Universal-3.5 Pro Realtime does streaming diarization with revision for up to 10 speakers — useful for multi-party rooms. For deployment, the plugin runs anywhere your LiveKit agent runs; migrate any host-specific deploy steps (e.g., a Fly.io walkthrough) from your existing tutorial content here and update model ids to universal-3-5-pro.

Build it

Talk to a live agent to hear it in action, or get your free API key and drop Universal-3.5 Pro Realtime into your LiveKit agent.

Frequently asked questions

Does AssemblyAI work with LiveKit out of the box?

Yes. AssemblyAI is a native streaming STT plugin in LiveKit Agents, running Universal-3.5 Pro Realtime, and it supports Context Carryover out of the box — no custom integration required.

Which turn-detection setting should I use?

Use LiveKit's English turn-detection model. Do not set turn_detection="stt" with the Pro model — that combination is not the recommended path.

Which model and endpoint does the plugin use?

Universal-3.5 Pro Realtime (speech_model=universal-3-5-pro) over wss://streaming.assemblyai.com/v3/ws. Streaming is $0.45/hr base; the managed Voice Agent API is a flat $4.50/hr.

How do I pass conversation context?

Use agent_context: seed it at connection time and update it after each agent reply. It cut WER by 10.2% across 20,000 voice-agent files, and prior finalized user turns carry over automatically.

How accurate is it for voice agents?

6.99% WER on the Pipecat voice-agent benchmark (versus Deepgram Flux at 15.58%), and it's the only model in Coval's independent Human Parity Zone at 3.40% WER and ~110ms p50.