Build a voice agent with LiveKit and AssemblyAI’s Voice Agent API

Here's the honest version of what building a voice agent used to cost you: a WebRTC stack you didn't want to maintain, plus a speech-to-text vendor, plus an LLM, plus a text-to-speech engine, plus the glue code that keeps all four in sync while a human is mid-sentence. Four integrations, three of which have nothing to do with your product.

This tutorial cuts that to two. LiveKit — our partner, and the easiest way to ship production-grade real-time audio — handles transport. AssemblyAI's Voice Agent API handles the entire AI pipeline over a single WebSocket. Your worker is a bridge between them, and it's about 250 lines of Python.

You'll end up with a multi-user, browser-ready voice agent you can talk to in a real LiveKit room, with barge-in, tool calling, and neural turn detection working out of the box. Let's get you there fast, then tune it.

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms." — David Zhao, Co-founder at LiveKit

Prefer the full framework-and-deploy path? If you want the LiveKit Agents framework with separate STT, LLM, and TTS plugins and a walkthrough of deploying to LiveKit Cloud, read our deeper guide: Build and deploy real-time AI voice agents using LiveKit and AssemblyAI. This post is the leaner quickstart — one WebSocket does all the AI.

The idea: LiveKit for transport, one WebSocket for the AI

WebRTC and Voice AI are different problems, and they have different best-in-class answers.

• LiveKit is the transport layer. SDKs for Web, iOS, Android, React Native, Flutter, and Unity. Built-in recording, simulcast, adaptive bitrate, and end-to-end encryption. A managed cloud and a self-hostable open-source server. It's the fastest way to get real audio in and out of rooms without building signaling and media infrastructure yourself.
• The Voice Agent API is the AI layer. One WebSocket gives you Universal-3.5 Pro Realtime for speech-to-text, an LLM that decides what to say, a TTS engine with 30+ voices, plus neural turn detection, barge-in, and tool calling — all server-side, all in one connection. Flat $4.50/hr, roughly 1 second end-to-end latency, and session resumption if a client drops and reconnects within 30 seconds.

Universal-3.5 Pro Realtime is the piece I'd point to first. It posts a 6.99% pooled word error rate on Pipecat's open STT benchmark of real agent conversations — not clean read-aloud audio, but the messy back-and-forth your agent will actually hear. It also takes the agent's own question as input (agent_context), which is a bigger deal than it sounds: short replies like "yeah, the second one" resolve far better when the model knows what was just asked. In our testing that context cut WER by 10.2% across 20,000 files.

This is the "invisible infrastructure" framing, and it's deliberate. You don't want your users to notice the speech pipeline any more than they notice TCP. Universal-3.5 Pro Realtime, the LLM, and the TTS engine are meant to disappear into a conversation that just works. Your job is the product on top; our job is to make the layer underneath boring in the best way.

When to reach for the LiveKit Agents framework instead

LiveKit ships an excellent agents framework, and it's the right tool for a real set of jobs. Reach for it when you want to orchestrate multiple specialized models yourself — a specific LLM here, a niche TTS voice there — or when you're already deep in the LiveKit Agents runtime and want AssemblyAI slotted in purely as the STT plugin. That's a supported, first-class path, and our full-deploy guide walks through it end to end.

Reach for the approach in this post when you'd rather not run an STT/LLM/TTS orchestration layer at all. One WebSocket, server-side, tuned as a unit. Both are valid. This one is just less to hold.

How this differs from the LiveKit Agents framework

‍

If you want the framework approach with AssemblyAI as the STT — the multi-plugin, deploy-to-LiveKit-Cloud path — start with our full LiveKit deploy guide. If you want one WebSocket to do all of the AI, keep reading.

Architecture

Browser / iOS / Android client (any LiveKit SDK)
        │
        │  WebRTC into a LiveKit room
        ▼
LiveKit Cloud (or self-hosted livekit-server)
        │
        │  remote audio track (24 kHz mono PCM16, after AudioStream resample)
        ▼
This Python worker — joins the room with livekit-rtc as a server-side participant
        │
        │  base64-encoded PCM16 chunks as { type: "input.audio", audio: "..." }
        ▼
┌────────────────────────────────────────────────────────────────┐
│  wss://agents.assemblyai.com/v1/ws                             │
│                                                                │
│  AssemblyAI Voice Agent API                                    │
│  ├── Universal-3.5 Pro Realtime  (speech → text)               │
│  ├── LLM                         (text → reply)                │
│  └── TTS                         (reply → 24 kHz PCM16 audio)  │
│                                                                │
│  + neural turn detection                                       │
│  + barge-in                                                    │
│  + tool calling                                                │
└────────────────────────────────────────────────────────────────┘
        │
        │  base64-encoded PCM16 as { type: "reply.audio", data: "..." }
        ▼
This Python worker — capture_frame() into rtc.AudioSource
        │
        │  published as a local audio track on the worker's participant
        ▼
LiveKit Cloud
        │
        │  WebRTC out
        ▼
Browser / iOS / Android client(s) hear the agent

The worker never touches STT, LLM, or TTS internals. It moves PCM16 in one direction and PCM16 in the other. The AI layer is opaque to the transport layer on purpose — that's the whole point.

Prerequisites

• Python 3.10+
• An AssemblyAI API key — free tier available, no credit card.
• A LiveKit server — either a free LiveKit Cloud project or a self-hosted livekit-server.
• A LiveKit client to talk to the agent. The fastest path is the hosted LiveKit Agents Playground.

You don't need a microphone or speakers on the worker machine. The worker is a server-side participant — all audio I/O happens in the browser or mobile client.

The complete code is in the companion repo: github.com/kelsey-aai/voice-agent-livekit.

Quick start

1. Clone and install

git clone https://github.com/kelsey-aai/voice-agent-livekit
cd voice-agent-livekit

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

2. Configure environment

cp .env.example .env

Fill in .env:

ASSEMBLYAI_API_KEY=           # https://www.assemblyai.com/dashboard/signup
LIVEKIT_URL=wss://<project>.livekit.cloud
LIVEKIT_API_KEY=              # LiveKit Cloud → Settings → Keys
LIVEKIT_API_SECRET=
ROOM_NAME=voice-agent-demo

For self-hosted LiveKit, run livekit-server --dev and use LIVEKIT_URL=ws://localhost:7880. The dev server prints an API key and secret on startup.

3. Run the worker

python worker.py

You should see the worker connect to LiveKit and wait for a participant:

Connecting to LiveKit room 'voice-agent-demo' at wss://...
Connected as worker. Publishing reply track …
Waiting for a participant to publish a microphone track …

4. Connect a client

The fastest way is the LiveKit Agents Playground:

1. Open the playground.
2. Paste your LIVEKIT_URL and a token. Generate a token from the LiveKit Cloud dashboard (Settings → Keys → Create token), set the room to voice-agent-demo and the identity to anything other than voice-agent.
3. Click Connect, allow microphone access, and start talking. You'll hear the agent reply through your browser.

You can also connect from your own LiveKit Web, iOS, Android, or Flutter app pointed at the same room.

How it works

The worker is one file (worker.py), roughly 250 lines. Six steps do the actual work.

1. Mint a LiveKit token and join the room

from livekit import api, rtc

token = (
    api.AccessToken(LIVEKIT_API_KEY, LIVEKIT_API_SECRET)
    .with_identity("voice-agent")
    .with_grants(api.VideoGrants(
        room_join=True, room=ROOM_NAME,
        can_publish=True, can_subscribe=True,
    ))
    .to_jwt()
)

room = rtc.Room()
await room.connect(LIVEKIT_URL, token)

AccessToken from livekit-api builds a signed JWT with the grants this worker needs: subscribe to incoming audio, publish a reply track. room.connect() opens the WebRTC signaling and media path.

2. Publish a local audio track for the agent's voice

audio_source = rtc.AudioSource(sample_rate=24_000, num_channels=1)
local_track = rtc.LocalAudioTrack.create_audio_track("agent-voice", audio_source)

await room.local_participant.publish_track(
    local_track,
    rtc.TrackPublishOptions(source=rtc.TrackSource.SOURCE_MICROPHONE),
)

AudioSource is LiveKit's pump for sending audio into a room. We configure it at 24 kHz mono — the Voice Agent API's default audio/pcm format — so we can hand reply audio straight to it without resampling.

3. Subscribe to the user's audio track

@room.on("track_subscribed")
def on_track_subscribed(track, publication, participant):
    if track.kind == rtc.TrackKind.KIND_AUDIO:
        asyncio.create_task(bridge_to_voice_agent(track))

LiveKit emits track_subscribed when a remote participant publishes a track and it gets routed to us. We only care about audio.

4. Forward microphone audio to the Voice Agent API

stream = rtc.AudioStream.from_track(
    track=mic_track,
    sample_rate=24_000,    # ask LiveKit to resample to 24 kHz for us
    num_channels=1,
)

async for event in stream:
    pcm16_bytes = bytes(event.frame.data)
    await ws.send(json.dumps({
        "type": "input.audio",
        "audio": base64.b64encode(pcm16_bytes).decode("ascii"),
    }))

AudioStream does the resampling for us. WebRTC carries audio at 48 kHz internally, but we ask for 24 kHz mono and the LiveKit FFI resampler handles the conversion. Each AudioFrame exposes data as a memoryview of int16 samples — converting to raw PCM16 bytes is just bytes(event.frame.data). Base64-encode and ship it as input.audio.

5. Play the agent's reply back into the room

elif t == "reply.audio":
    pcm = base64.b64decode(event["data"])
    samples = len(pcm) // 2  # 2 bytes per int16, mono
    frame = rtc.AudioFrame(
        data=pcm,
        sample_rate=24_000,
        num_channels=1,
        samples_per_channel=samples,
    )
    await audio_source.capture_frame(frame)

The agent streams reply.audio events as soon as the LLM begins generating — you don't wait for the whole reply. Each chunk is wrapped in an AudioFrame and pushed into the AudioSource, which queues it up to 1 second deep and drains at 24 kHz on its own clock. That streaming behavior is a big part of why end-to-end latency lands around a second.

6. Handle barge-in

When the user speaks while the agent is talking, two things can happen — and both flush the playback queue:

elif t == "input.speech.started":
    # User started talking; stop playback so they can be heard.
    audio_source.clear_queue()

elif t == "reply.done":
    if event.get("status") == "interrupted":
        audio_source.clear_queue()

AudioSource.clear_queue() immediately discards every queued frame so the user doesn't hear stale agent audio after they've spoken over it. Without this, the AudioSource would keep playing the buffered ~1 second of TTS even after the server has cut the agent off.

Tuning the agent

Pick a voice

"output": {"voice": "james"}     # conversational US male
"output": {"voice": "sophie"}    # clear UK female
"output": {"voice": "diego"}     # Latin American Spanish
"output": {"voice": "arjun"}     # Hindi/Hinglish, code-switches with English

The full catalog is in the voices reference. Multilingual voices code-switch automatically. The Voice Agent API currently supports six languages end to end: English, Spanish, French, German, Italian, and Portuguese.

Adjust the system prompt and greeting

These are the two highest-leverage settings for how the agent feels:

"session": {
    "system_prompt": (
        "You are a customer support agent for Acme. Speak in 1-2 short "
        "sentences. Confirm the user's question before answering."
    ),
    "greeting": "Hi, this is Acme support — what's going on?",
}

You can re-send session.update mid-conversation to swap the prompt or voice. greeting is locked once spoken, but system_prompt and voice are not.

Tune turn detection

Default values work for most apps. Override anything you want under session.input.turn_detection:

"input": {
    "turn_detection": {
        "vad_threshold": 0.5,        # 0.0-1.0; higher = ignore more background noise
        "min_silence": 600,          # ms before confident end-of-turn
        "max_silence": 1500,         # ms hard ceiling
        "interrupt_response": True,  # set False to disable barge-in entirely
    }
}

For deliberate speech (eldercare, healthcare intake), raise max_silence to 2500. For fast-paced conversation, drop min_silence to 300.

Boost domain-specific terms and tune for the room

If your conversation includes product names, medical terms, or rare proper nouns, add them to session.input.keyterms. The recognizer biases toward those words, which dramatically improves accuracy on domain vocabulary:

"input": { "keyterms": ["Universal-3.5 Pro Realtime", "AssemblyAI", "LiveKit"] }

Two more accuracy levers come from the underlying Universal-3.5 Pro Realtime streaming model. Unlike keyterms, they aren't fields you set in the Voice Agent API session config — on the Voice Agent API they're applied automatically, and they're exposed as direct parameters only on the raw Streaming Speech-to-Text API:

• voice_focus for noisy, far-field audio. LiveKit rooms often carry laptop-mic or open-office audio with background chatter and HVAC. Enabling voice_focus isolates the speaker's voice before it hits the recognizer, which is exactly what you want when the participant isn't wearing a headset. Turn it on for anything approaching far-field capture.
• agent_context for short replies. Pass the agent's own last question along with the audio so Universal-3.5 Pro Realtime can resolve terse answers. "Uh, the blue one" and "yes, transfer it" are ambiguous in isolation but obvious given the question that preceded them. This is the same agent_context behavior that cut WER by 10.2% across 20,000 files in our testing, and it's free accuracy in a back-and-forth conversation.

There's also Context Carryover, a rolling conversation memory that keeps recent turns in view so the recognizer stays sharp as the dialogue develops. If you're using the LiveKit Agents framework path instead of this transport-only approach, the context-carryover parameters landed in the LiveKit Agents SDK in v1.6.0 — use the latest SDK (v1.6.5) to pick them up. [VERIFY: confirm current SDK version]

Multiple participants in one room

This worker bridges one remote audio track to the Voice Agent API. The agent only listens to the first participant who publishes audio — additional participants are ignored. There are two ways to scale this:

1. One agent per room. Spin up a separate worker process per room (LiveKit Cloud's worker dispatch makes this easy). Best for one-on-one use cases like phone-style support agents.
2. Mix participants before sending. If you genuinely want a meeting-style multi-talker agent, mix all remote audio with rtc.AudioMixer (bundled in livekit-rtc) and send the mix to one Voice Agent API session. The mixer handles sample-rate alignment and frame timing for you.

For most real-world apps, option 1 is the right call.

Troubleshooting

The worker connects but the client never hears the agent. The local track is published before the user joins — that's correct — but make sure your client subscribed to it. In the LiveKit Agents Playground, the agent's track shows up under the participant identity voice-agent. Confirm can_subscribe=True on the client's token.

UNAUTHORIZED close on the AssemblyAI WebSocket. Your ASSEMBLYAI_API_KEY is missing, expired, or pasted with whitespace. Grab a fresh key from the AssemblyAI dashboard and confirm .env is loaded.

LiveKit ConnectError: invalid token. The JWT signature didn't validate against the LIVEKIT_API_SECRET. Check that the URL, key, and secret all come from the same LiveKit project. For self-hosted, regenerate keys with livekit-server generate-keys.

Audio is choppy or robotic. Almost always the audio buffer running dry. Confirm the worker is reaching the Voice Agent API with low jitter — run python worker.py close to your network egress. Inside AudioSource(... queue_size_ms=1000) you have one second of headroom; raise it to 2000 if you're seeing transient stalls.

Audio sounds pitched up or down. Sample-rate mismatch. Both AudioSource and AudioStream.from_track must be configured at sample_rate=24_000, num_channels=1. The default for AudioStream is 48 kHz — make sure you override it.

Agent keeps interrupting itself. Browser clients with getUserMedia({ audio: { echoCancellation: true } }) (the default) handle this automatically. If you see it on a custom mobile client, make sure AEC is enabled on the capture side. The Voice Agent API also exposes session.input.turn_detection.interrupt_response = false to disable barge-in entirely while you debug.

The full Voice Agent API troubleshooting guide is in the docs.

Try it, then build the real thing

You've got a working voice agent in a real LiveKit room. Barge-in works, turn detection works, and you never wrote a line of STT, LLM, or TTS orchestration. That's the version of this that should feel a little too easy.

Here's the insight worth leaving with: the reason this collapses to two integrations isn't that LiveKit and AssemblyAI happen to fit together. It's that "transport" and "the AI" are the only two boundaries in a voice agent that actually deserve to be separate systems. Everything else — endpointing, barge-in, the STT/LLM/TTS handoff — only got split across vendors for historical reasons, and every seam between them is a place latency and bugs hide. Putting the whole AI pipeline behind one WebSocket doesn't just save you keys; it removes the seams. When you're deciding how to architect your next agent, that's the question to ask about any component: is this a real boundary, or an accident of who sold it to you?

Related reading

• Voice Agent API product overview
• Universal-3.5 Pro Realtime
• Build a Python voice agent with the Voice Agent API
• Build and deploy real-time AI voice agents using LiveKit and AssemblyAI (the deeper, full-deploy framework guide)
• Streaming Speech-to-Text
• What are AI voice agents?
• Pricing
• Voices catalog

Frequently asked questiosn

How do I build a voice agent with LiveKit?

Use LiveKit for transport and AssemblyAI's Voice Agent API for the AI. Join a LiveKit room as a server-side participant with the livekit-rtc Python SDK, forward the user's microphone audio (24 kHz mono PCM16) to the Voice Agent API over one WebSocket, and publish the agent's reply.audio back into the room as a local track. The full working worker is about 250 lines — clone it from github.com/kelsey-aai/voice-agent-livekit and follow the quick start above. If you'd rather use the LiveKit Agents framework with separate STT/LLM/TTS plugins, see our full LiveKit deploy guide.

Does AssemblyAI integrate with LiveKit?

Yes. LiveKit is an active AssemblyAI partner, and there are two supported ways to combine them. First, the transport-only approach in this tutorial: LiveKit moves the audio, and the Voice Agent API handles speech-to-text, the LLM, and text-to-speech over a single WebSocket. Second, the LiveKit Agents framework approach, where AssemblyAI's Universal-3.5 Pro Realtime slots in as the STT plugin and you configure the LLM and TTS separately. There's a drop-in AssemblyAI plugin for LiveKit either way.

Voice Agent API vs Retell — what's the difference?

Different layers of the stack. Retell is a voice agent platform — a hosted product with its own dashboard, agent builder, and opinionated workflow. AssemblyAI is infrastructure: the Voice Agent API is a single WebSocket that gives you speech-to-text, an LLM, TTS, turn detection, barge-in, and tool calling, which you compose into your own product however you want. If you want a managed platform to configure agents in a UI, that's a platform decision. If you're a developer building your own agent and you want a fast, accurate, flat-rate ($4.50/hr) pipeline you control over an API — with drop-in plugins for LiveKit and Pipecat — that's this. You can also run the Voice Agent API behind LiveKit, Twilio, Agora, or Daily transport, because it doesn't care where the audio comes from.

Is this the LiveKit Agents framework?

No. The LiveKit Agents framework expects you to plug in separate STT, LLM, and TTS components. This tutorial uses the LiveKit livekit-rtc Python SDK directly to join a room as a server-side participant, then forwards audio to the AssemblyAI Voice Agent API, which replaces all three. If you want the framework approach with AssemblyAI as the STT, start with the full LiveKit deploy guide.

What speech-to-text model powers the Voice Agent API?

Universal-3.5 Pro Realtime, released June 23, 2026. It posts a 6.99% pooled word error rate on Pipecat's open STT benchmark of real agent conversations, takes the agent's question as input via agent_context (which cut WER by 10.2% across 20,000 files), keeps a rolling conversation memory with Context Carryover, and includes voice_focus for isolating a speaker's voice in noisy rooms. See the Universal-3.5 Pro Realtime announcement for the full breakdown.

What audio format does the Voice Agent API expect?

By default the Voice Agent API uses audio/pcm — 16-bit signed little-endian PCM at 24,000 Hz, mono, base64-encoded. This worker configures both the LiveKit AudioStream (incoming) and AudioSource (outgoing) at 24 kHz mono so no manual resampling is needed; LiveKit's native FFI resampler handles the conversion to and from WebRTC's internal 48 kHz. For telephony you can switch to audio/pcmu (G.711 μ-law, 8 kHz) under session.input.format and session.output.format.

Can the Voice Agent API call tools from inside a LiveKit room?

Yes. Register tool definitions in session.tools on session.update. When the agent decides to invoke one, the server emits a tool.call event with a call_id, function name, and arguments. Run the tool in your worker, then send back a tool.result event after you receive the next reply.done (not immediately on tool.call). Tool calls work the same whether the audio is coming from sounddevice, a LiveKit room, or a phone line — the transport is irrelevant to the AI layer.

How do I scale to many concurrent rooms?

Run one worker per room. LiveKit Cloud's agent dispatch (or your own room-watching service) can spin up a worker per active room, and each worker holds one Voice Agent API WebSocket. The Voice Agent API charges per session at a flat $4.50/hr, LiveKit charges per minute of WebRTC, and both scale horizontally — there's no shared coordination point.

How much does the Voice Agent API cost?

The Voice Agent API is a flat $4.50/hr, and there's a free tier so you can build and test without a credit card. For current pricing details, see the AssemblyAI pricing page and the LiveKit Cloud pricing page.

‍