Build a voice agent with LiveKit

Voice agents are no longer demo territory. Developers are shipping them to production — for customer support, sales coaching, medical intake, and anything else where a keyboard gets in the way. The challenge isn't the idea. It's wiring up the pipeline without it becoming a second full-time job.

This tutorial walks you through building a complete voice agent using LiveKit Agents as your orchestration framework and AssemblyAI's Universal-3.5 Pro Realtime model as the speech-to-text layer. You'll wire in OpenAI GPT-4o for the LLM and Cartesia for text-to-speech, run it locally first, and then connect it to the LiveKit Agents Playground.

If you've never touched LiveKit before, that's fine. We'll cover what you need to know.

What is LiveKit?

LiveKit is an open-source real-time communication platform built on top of WebRTC. It handles the hard parts of real-time audio and video — signaling, media routing, scaling — so you can focus on your application logic instead.

At the center of every LiveKit application is a room. Participants join rooms and publish audio or video tracks. Other participants subscribe to those tracks. A LiveKit server (self-hosted or via LiveKit Cloud) acts as the Selective Forwarding Unit that routes media between participants without requiring peer-to-peer connections at scale.

LiveKit Agents is the framework layer built on top of this infrastructure specifically for AI-powered voice and video agents. It handles orchestration between STT, LLM, and TTS services — you configure the components, and LiveKit Agents manages the flow. Your agent joins a room like any other participant, listens to audio tracks, and responds.

The result: you write agent logic, not media plumbing.

The voice agent pipeline

Voice agents follow a three-step cascading pipeline:

1. Speech-to-text (STT): The user speaks. Audio streams into your STT model, which returns a transcript in real time.
2. LLM: The transcript goes to a language model, which generates a response.
3. Text-to-speech (TTS): The LLM's response gets converted to audio and streamed back to the user.

Here's what that looks like in this stack:

WebRTC (LiveKit room)
        ↓
  LiveKit Cloud
        ↓
  AssemblyAI Universal-3.5 Pro Realtime  ← speech-to-text
        ↓ transcript + turn signal
  OpenAI GPT-4o                          ← LLM
        ↓ text response
  Cartesia Sonic                          ← text-to-speech
        ↓ audio
  Back to LiveKit room

The key variable in this pipeline is the STT layer. If the transcript is wrong, the LLM responds to the wrong input. That's why the STT choice matters more than most people realize — and it's why we're using Universal-3.5 Pro Realtime here instead of the default.

Why Universal-3.5 Pro Realtime for STT?

The LiveKit AssemblyAI plugin supports multiple models. The default is universal-streaming — fast and solid for English-only use cases. But for production voice agents, you want universal-3-5-pro, AssemblyAI's flagship real-time model and the new default for voice AI.

The difference shows up in three places: accuracy, turn detection, and conversation context.

Accuracy

Real voice agent conversations involve email addresses, phone numbers, account numbers, URLs — the kind of structured entities that most STT models routinely mangle. Universal-3.5 Pro Realtime was built for exactly this. On Pipecat's open STT benchmark of real agent conversations, it posts a market-leading 6.99% word error rate:

It leads on time-to-first-token in third-party latency benchmarks as well, delivering partial and final transcripts at roughly 150 ms P50.

Turn detection

Universal-3.5 Pro Realtime uses punctuation-based end-of-turn detection: after a short pause, the model checks for terminal punctuation (. ? !) to decide whether the turn is actually complete, rather than firing on silence alone the way plain VAD does. The result is fewer false triggers on mid-sentence pauses, faster responses when someone finishes, and a conversation that doesn't feel like it's fighting you. You steer the trade-off with a single mode preset (min_latency, balanced, max_accuracy) and fine-tune with min_turn_silence / max_turn_silence.

Conversation context

This is the newest edge, and it's unique to the U3 Pro family. With Context Carryover, the model hears each user turn in the context of what your agent just said. When your agent asks *"What's your email address?"*, the model can produce user@assemblyai.com instead of user at assemblyai dot com. Feeding the agent's question to the model cut WER by 10.2% across a 20,000-file voice-agent benchmark. In LiveKit, this is a one-line toggle (agent_context_carryover=True) — more below.

At $0.45/hr, it's the STT layer your voice agent should be running on.

Prerequisites

• Python 3.11+
• A microphone and speakers (for local testing)
• API keys for:
• AssemblyAI — free tier available at assemblyai.com
• LiveKit Cloud — free tier available at livekit.io
• OpenAI
• Cartesia

Step 1: Installation

Create a virtual environment and install the dependencies:

python -m venv .venv
source .venv/bin/activate   # On Windows: .venv\Scripts\activate

pip install "livekit-agents[assemblyai,silero,codecs]>=1.6" python-dotenv

You'll also need plugins for your LLM and TTS providers. For GPT-4o and Cartesia:

pip install "livekit-agents[openai,cartesia]>=1.6"

One note: Universal-3.5 Pro Realtime, agent_context, and Voice Focus require livekit-agents 1.6+ (1.6.5+ for automatic context carryover). On older versions the universal-3-5-pro model ID throws a validation error — run pip install --upgrade "livekit-agents>=1.6" if you hit that.

Step 2: Configure your API keys

Create a .env file in your project root:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
ASSEMBLYAI_API_KEY=your_assemblyai_key
OPENAI_API_KEY=your_openai_key
CARTESIA_API_KEY=your_cartesia_key

Get your AssemblyAI API key from the AssemblyAI dashboard under API Keys. Get your LiveKit URL, API key, and secret from your LiveKit Cloud project settings.

Step 3: Build the agent

Create a file called agent.py and add the following:

from dotenv import load_dotenv
from livekit import agents
from livekit.agents import AgentSession, Agent, TurnHandlingOptions
from livekit.plugins import (
    assemblyai,
    cartesia,
    openai,
    silero,
)

load_dotenv()


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="You are a helpful voice AI assistant."
        )


async def entrypoint(ctx: agents.JobContext):
    await ctx.connect()

    session = AgentSession(
        stt=assemblyai.STT(
            model="universal-3-5-pro",
            min_turn_silence=100,    # Silence (ms) before a speculative
end-of-turn check
            max_turn_silence=1000,   # Override the plugin default of 100 when
turn_detection="stt"
            vad_threshold=0.3,       # Match Silero's activation_threshold below
        ),
        llm=openai.LLM(model="gpt-4o"),
        tts=cartesia.TTS(),
        vad=silero.VAD.load(
            activation_threshold=0.3,
        ),
        turn_handling=TurnHandlingOptions(
            turn_detection="stt",       # Use AssemblyAI's end-of-turn signal
directly
            endpointing={"min_delay": 0},  # Avoid additive delay in STT mode
        ),
    )

    await session.start(
        room=ctx.room,
        agent=Assistant(),
    )

    await session.generate_reply(
        instructions="Greet the user and offer your assistance."
    )


if __name__ == "__main__":
    agents.cli.run_app(
        agents.WorkerOptions(entrypoint_fnc=entrypoint)
    )

Let's walk through what's happening.

The Assistant class

Agent is where you define your agent's personality and behavior. The instructions string is the system prompt — it goes directly to the LLM. This is where you'd define your agent's persona, rules, and any context it needs.

The AgentSession

This is the core of your agent. You're wiring together four components:

• stt: AssemblyAI's Universal-3.5 Pro Realtime model. We set model="universal-3-5-pro" explicitly, along with turn-detection parameters (more below).
• llm: OpenAI GPT-4o. Swap the model string to use GPT-4o mini or any other OpenAI-compatible model.
• tts: Cartesia. The default voice works well out of the box.
• vad: Silero VAD, for interruption handling. activation_threshold is set to 0.3 to match Universal-3.5 Pro Realtime's internal VAD threshold — keeping them aligned avoids a dead zone where one system thinks the user is speaking and the other doesn't.

Turn detection with turn_detection="stt"

Setting turn_detection="stt" tells LiveKit Agents to use Universal-3.5 Pro Realtime's own end-of-turn signal instead of LiveKit's turn detector. This is the recommended mode — it's what the model was built for. Pair it with endpointing={"min_delay": 0} so you don't stack delay on top of the STT's already-fast response, and set max_turn_silence=1000 explicitly (the plugin defaults it to 100, which is often too aggressive in this mode).

The entrypoint function

This runs when a new job comes in. ctx.connect() joins the LiveKit room. Then we start the session and fire off an initial greeting. The agent keeps running until the room closes or the session ends.

Step 4: Run it

There are two modes for running locally.

Console mode

Console mode lets you talk to the agent directly from your terminal — no LiveKit Cloud connection needed. It's the fastest way to test.

python agent.py console

Speak into your microphone. You'll see transcripts printed in real time, and the agent's responses play through your speakers.

Dev mode

Dev mode connects your agent to LiveKit Cloud and makes it available through the Agents Playground:

python agent.py dev

Then open agents-playground.livekit.io, enter your LiveKit URL and API key, and click Connect.

Tuning turn detection

Universal-3.5 Pro Realtime's turn detection is punctuation-based, controlled by two silence parameters and a high-level mode preset. The defaults work well for general conversational agents; tune from there.

stt=assemblyai.STT(
    model="universal-3-5-pro",
    mode="balanced",       # "min_latency" (fastest) · "balanced" · "max_accuracy"
    min_turn_silence=100,  # ms of silence before the speculative end-of-turn check
    max_turn_silence=1000, # hard ceiling: turn ends after this much silence regardless
)

How to think about each one:

• mode: the highest-level dial. min_latency for snappy back-and-forth, max_accuracy for entity-heavy or deliberate speech. It sets sensible defaults for the levers below.
• min_turn_silence: how long the model waits before checking for terminal punctuation. Raise it (e.g. to 400 ms) when brief pauses split entities like john.smith@gmail.com across turns.
• max_turn_silence: the hard ceiling. Raise to 2000 ms+ for healthcare intake or users who speak more deliberately, so the turn isn't cut off mid-thought.

Note: end_of_turn_confidence_threshold does not apply to the U3 Pro family — that parameter belongs to the older universal-streaming models. Universal-3.5 Pro Realtime uses punctuation-based detection instead.

You can also update these mid-session without reconnecting — useful when the conversation shifts context:

# Widen the window while the user dictates an address or account number
stt.update_options(max_turn_silence=3000)

# Later, reset to baseline
stt.update_options(max_turn_silence=1000)

Conversation context (Context Carryover)

Universal-3.5 Pro Realtime keeps a short, per-session memory of the dialog and can take your agent's most recent reply as context for the next user turn. With the agent's question in context, it anticipates the answer, sharpens entity recognition, and disambiguates similar-sounding words — the biggest wins are on short replies ("yes", "7pm", single names) and spelled-out entities.

In LiveKit, turn it on with one flag; the plugin forwards each agent reply automatically after every turn:

session = AgentSession(
    stt=assemblyai.STT(
        model="universal-3-5-pro",
        agent_context_carryover=True,   # Requires livekit-agents 1.6.5+
    ),
    # llm=your_llm_plugin(),
    # tts=your_tts_plugin(),
)

No event wiring, no reconnects. This is the single highest-leverage accuracy setting for a voice agent, and it's unique to the U3 Pro family.

Enabling keyterm prompting

If your agent handles domain-specific vocabulary — product names, medical terms, financial jargon, brand names — boost recognition with keyterm prompting. It takes effect immediately, no restart:

# After session.start():
await session.stt.update_options(
    keyterms_prompt=["AssemblyAI", "Universal-3.5 Pro Realtime", "LiveKit"]
)

Especially useful for structured data like account numbers, medication names, or product SKUs — exactly the entities generic STT models get wrong most often. Keyterm prompting is included on Universal-3.5 Pro Realtime at no extra cost.

Handling noisy audio with Voice Focus

For calls from cars, kiosks, call-center floors, or laptop mics, enable Voice Focus — server-side noise suppression that isolates the primary speaker before audio reaches the model. Use near-field for close-talking mics (headsets, handsets) and far-field for distant capture (rooms, kiosks):

stt=assemblyai.STT(
    model="universal-3-5-pro",
    voice_focus="far-field",   # "near-field" for close-talking mics
)

Prefer this to running your own client-side noise cancellation, which tends to introduce artifacts that hurt accuracy more than the noise itself.

Swapping components

One of the cleaner aspects of LiveKit Agents is how easy it is to swap providers. The abstract interfaces for STT, LLM, and TTS are consistent — you change the plugin, not the surrounding code.

To swap the LLM from GPT-4o to Claude:

from livekit.plugins import anthropic

session = AgentSession(
    stt=assemblyai.STT(model="universal-3-5-pro"),
    llm=anthropic.LLM(model="claude-sonnet-5"),
    tts=cartesia.TTS(),
)

To swap TTS from Cartesia to ElevenLabs:

from livekit.plugins import elevenlabs

tts=elevenlabs.TTS(voice_id="your-voice-id")

The STT layer is the one component you don't want to casually swap — that's where your accuracy comes from. The rest of the pipeline is genuinely modular.

Next steps

You now have a working voice agent running Universal-3.5 Pro Realtime. A few directions from here:

• Deploy to LiveKit Cloud: Run python agent.py start to register your agent as a persistent worker.
• Add tool calling: Give your agent the ability to look things up, take actions, or connect to your backend through the LLM layer.
• Enable speaker labels: For multi-party conversations, Universal-3.5 Pro Realtime supports real-time speaker labels as a per-session toggle (speaker_labels=True).
• Add telephony: LiveKit has native SIP support — pair it with Telnyx or Twilio and your agent is live on a phone number.

The full parameters reference for Universal-3.5 Pro Realtime on LiveKit is at assemblyai.com/docs/voice-agents/livekit-u3-rt-pro.

Frequently asked questions

What is LiveKit Agents and how does it work for building voice agents?

LiveKit Agents is a framework within the LiveKit platform for building AI-powered voice and video agents. It handles orchestration between a speech-to-text model, an LLM, and a text-to-speech model — you configure the components and LiveKit manages the real-time audio routing, turn flow, and session lifecycle. Your agent joins a LiveKit room as a participant, listens to audio tracks published by users, runs audio through the STT → LLM → TTS pipeline, and streams audio responses back. It's built on LiveKit's WebRTC infrastructure and supports both self-hosted and LiveKit Cloud deployments.

What's the difference between Universal-3.5 Pro Realtime and Universal-Streaming for voice agents?

Universal-3.5 Pro Realtime (universal-3-5-pro) is AssemblyAI's flagship model, optimized for production voice agents; Universal-Streaming is the faster, lower-cost model for English-only use cases. The differences are accuracy, turn detection, and context: Universal-3.5 Pro Realtime uses punctuation-based end-of-turn detection, supports Context Carryover (feeding the agent's question to the model for sharper answers), and delivers markedly better accuracy on structured entities like emails, phone numbers, and account numbers. It costs $0.45/hr and supports 18 languages with mid-sentence code-switching; Universal-Streaming is $0.15/hr and English-only.

What is the best streaming speech-to-text API for building voice agents with LiveKit?

AssemblyAI's Universal-3.5 Pro Realtime is the recommended choice for LiveKit voice agents. It has a native one-line integration via the assemblyai plugin (model="universal-3-5-pro" on livekit-agents 1.6+), leads Pipecat's open real-agent benchmark at 6.99% WER, delivers partial and final transcripts around 150 ms P50, and exposes end-of-turn detection that LiveKit uses directly with turn_detection="stt". It also adds Context Carryover, which cut WER 10.2% on a 20,000-file voice-agent benchmark.

How does turn detection in Universal-3.5 Pro Realtime work, and why does it matter?

Universal-3.5 Pro Realtime uses punctuation-based end-of-turn detection: after a short silence it checks for terminal punctuation (. ? !) to decide whether the speaker is actually finished, rather than simply waiting out a fixed silence window like plain VAD. That means fewer false triggers on mid-sentence pauses and snappier responses in natural conversation. You steer it with a mode preset (min_latency / balanced / max_accuracy) and tune min_turn_silence and max_turn_silence — and you can adjust them mid-session without reconnecting.

How much does it cost to build a voice agent using AssemblyAI Universal-3.5 Pro Realtime with LiveKit?

Universal-3.5 Pro Realtime is priced at $0.45/hr, billed on session duration with no minimum commitments and unlimited concurrent sessions. Keyterm prompting is included. You'll budget separately for your LLM (e.g. OpenAI GPT-4o) and TTS provider (e.g. Cartesia), which bill through their own providers. AssemblyAI offers a free tier to get started with no credit card required.

Can I swap out the LLM or TTS components in a LiveKit voice agent?

Yes — LiveKit Agents is designed around modular, swappable components. AgentSession takes separate stt, llm, and tts parameters, each backed by a consistent plugin interface. You can swap OpenAI GPT-4o for Anthropic Claude or any OpenAI-compatible model without changing the surrounding code; Cartesia can be replaced with ElevenLabs, OpenAI TTS, or any other supported provider. The STT layer is the one you shouldn't swap casually — a wrong transcript produces a wrong answer.