How to build with the Voice Agent API

Building a voice agent used to mean wiring together three or four vendors — a speech-to-text provider, an LLM, a text-to-speech engine, and some glue for turn detection — then spending weeks tuning the seams between them. The Voice Agent API collapses all of that into one WebSocket connection. You send audio in, you get audio back, and the whole pipeline — STT, LLM, TTS, neural turn detection, and tool calling — runs behind a single endpoint at a flat $4.50/hr.

This is a build guide, not a brochure. By the end you'll have a working agent that listens, thinks, talks, handles interruptions, and calls your own tools. Everything here is copy-paste runnable against the live API.

What is the Voice Agent API?

The Voice Agent API is a managed, full-stack path for building voice agents. One WebSocket connection to wss://agents.assemblyai.com/v1/ws handles the entire conversational loop: it transcribes the caller, runs your prompt through an LLM, synthesizes a reply, and streams that reply back as audio — while managing who's talking and when.

Here's the positioning that matters. AssemblyAI is invisible infrastructure. We're not the agent, and we're not a no-code agent platform with a ceiling you'll eventually hit. We're the Voice AI layer you build your agent on top of. That distinction shows up everywhere in the API design: everything is a standard JSON-over-WebSocket protocol, there's no proprietary SDK you're forced to adopt, and it works cleanly with coding agents like Claude Code, Cursor, and Windsurf.

Under the hood, the speech-to-text foundation is Universal-3.5 Pro Realtime — our streaming flagship. That's the same model you'd reach for if you built a bring-your-own stack directly on our streaming speech-to-text, except here it's already integrated and tuned for agent conversations.

Voice Agent API vs. bring-your-own-stack

There are two ways to build on AssemblyAI, and it's worth being explicit about when to choose which:

• Voice Agent API (the managed path). One WebSocket, flat $4.50/hr all-in, STT + LLM + TTS + turn detection handled for you. Start here. This is the fastest way to production and what most teams should use.
• Bring-your-own-stack. You wire Universal-3.5 Pro Realtime streaming ($0.45/hr base) to your own LLM and TTS, connecting the transport yourself. More control, more moving parts. This is the path our Agora and Daily.co integration guides walk through.

This guide covers the managed path. If you'd rather assemble the pieces yourself, the two partner guides above are your starting points.

Why build on the Voice Agent API?

Two reasons stand out: accuracy and price.

Accuracy, on real conversations. On the Pipecat open STT benchmark — which measures word error rate on actual voice-agent conversations, not clean read-aloud audio — Universal-3.5 Pro Realtime posts a 6.99% pooled WER. Here's how that compares:

The entity error rate column is the one to watch. When a caller says a phone number, an email, or a name, that's usually the whole point of the call — and it's exactly where weaker models fall apart. A 15.31% entity error rate against Deepgram's 50.5% is the difference between an agent that books the appointment and one that mishears the callback number. See the full methodology on our benchmarks page.

Predictable pricing. The Voice Agent API is a flat $4.50/hr, all-in, billed per second of session. That's roughly 4x cheaper than OpenAI's Realtime API, which lands around $18/hr once you account for its metered audio in and out. No per-character TTS fees, no pre-purchased concurrency tiers, no surprise line items. One number, and it covers STT, LLM, and TTS together.

Connect to the Voice Agent API

The handshake is three steps: open the WebSocket with Bearer auth, send a session.update config, and wait for session.ready before you start streaming audio.

Note the auth scheme — the Voice Agent API uses Authorization: Bearer YOUR_API_KEY. (Our REST and standalone streaming endpoints use a bare API key with no Bearer prefix; the Voice Agent API is the exception.)

# pip install websockets sounddevice numpy
import asyncio, base64, json, os
import sounddevice as sd
import websockets

URL = "wss://agents.assemblyai.com/v1/ws"
SAMPLE_RATE = 24_000  # Voice Agent audio is 24 kHz PCM16, base64 inside JSON

async def main():
    headers = {"Authorization": f"Bearer {os.environ['ASSEMBLYAI_API_KEY']}"}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "system_prompt": "You are a friendly support agent. Keep replies to one or two sentences.",
                "greeting": "Hi there, how can I help today?",
                "output": {"voice": "anna"},
            },
        }))

        async for raw in ws:
            event = json.loads(raw)
            if event["type"] == "session.ready":
                print(f"Session ready: {event['session_id']}")
                # Now start streaming mic audio as input.audio events
            elif event["type"] == "transcript.user":
                print(f"User: {event['text']}")
            elif event["type"] == "transcript.agent":
                print(f"Agent: {event['text']}")

asyncio.run(main())

A few details that trip people up:

• Audio format. Input and output audio are PCM16 mono at 24 kHz, base64-encoded inside JSON events — not raw binary frames. Send mic audio as {"type":"input.audio","audio":"<base64 PCM16 24kHz>"}.
• Reading the reply. The synthesized audio arrives in reply.audio events, and the payload sits in the data field (not audio). Write it straight to a 24 kHz output buffer.
• Order matters. Send session.update immediately, then wait for session.ready before sending any input.audio.
• Voices. Valid English voices include anna, alba, charles, eve, george, jane, jean, mary, and michael (US), plus paul and vera (UK). Don't use legacy names — they've been removed.
• Browser clients. Never ship your API key to the browser. Mint a temporary token and pass it as ?token=... instead.

Ready to build? Grab your free API key — new accounts get free API credit to start, no credit card required.

How neural turn detection works

This is the part most stacks get wrong, and it's where the Voice Agent API earns its keep.

Traditional voice pipelines detect end-of-turn with a voice activity detector (VAD) — essentially a silence timer. Someone stops making sound for N milliseconds, and the agent assumes they're done. The problem is obvious the first time a caller pauses mid-sentence to think: the agent barges in, and the conversation falls apart.

The Voice Agent API uses neural end-of-turn detection instead. It reads tonality and pacing — the acoustic cues a human uses to know when someone's actually finished a thought — and lands its decision in roughly 300ms. It's not waiting for silence; it's understanding the shape of the utterance. That's a fundamentally different mechanism than a VAD threshold, and it's why turn-taking feels natural rather than stilted.

You can still tune it. Explicitly setting min_silence or max_silence switches to a fixed-timer approach (and disables adaptive endpointing for the rest of the session):

import json

await ws.send(json.dumps({
    "type": "session.update",
    "session": {
        "input": {
            "turn_detection": {
                "vad_threshold": 0.5,
                "min_silence": 1500,
                "max_silence": 5000,
                "interrupt_response": True
            }
        }
    }
}))

‍

How interruption handling works

Interruptions are enabled by default. A multimodal model looks at both the incoming audio and the Universal-3.5 Pro Realtime transcript to tell the difference between backchanneling ("uh-huh," "right," "mm-hm") and an actual interruption. Backchannels don't stop the agent; real interruptions do.

When the caller genuinely interrupts, you'll get a reply.done event with status: "interrupted". Flush your playback buffer when you see it:

async for raw in ws:
    event = json.loads(raw)
    if event["type"] == "reply.done" and event.get("status") == "interrupted":
        # User actually interrupted. Flush playback buffer.
        flush_playback()

Update your agent mid-conversation

Most fields on session.update stay mutable after session.ready — including system_prompt, input.turn_detection, input.keyterms, output.volume, and tools. A common pattern: loosen turn detection right after the agent asks an open-ended question, then tighten it again once the caller starts answering.

baseline = {
    "vad_threshold": 0.5,
    "min_silence": 1000,
    "max_silence": 3000,
    "interrupt_response": True
}

waiting_for_answer = False

async def set_turn_detection(td):
    await ws.send(json.dumps({
        "type": "session.update",
        "session": {"input": {"turn_detection": td}}
    }))

async for raw in ws:
    event = json.loads(raw)
    if event["type"] == "transcript.agent" and event.get("text", "").rstrip().endswith("?"):
        waiting_for_answer = True
        await set_turn_detection({**baseline, "min_silence": 2200, "max_silence": 6000})
    elif event["type"] == "transcript.user" and waiting_for_answer:
        waiting_for_answer = False
        await set_turn_detection(baseline)

Add tool calling to your agent

Register tools in session.tools, and the agent will emit tool.call events when it wants to use one. You run the tool and hand back a tool.result. One important detail: tool definitions use a flat schema — {"type":"function","name":...,"parameters":{...}} — not OpenAI's nested form.

import json

last_event = None
pending_tools = []

async def flush_if_idle():
    if last_event != "reply.done" or not pending_tools:
        return
    for tool in pending_tools:
        await ws.send(json.dumps({
            "type": "tool.result",
            "call_id": tool["call_id"],
            "result": json.dumps(tool["result"]),  # must be a JSON-encoded string
        }))
    pending_tools.clear()

async for raw in ws:
    event = json.loads(raw)
    t = event.get("type")
    if t == "tool.call":
        # event["arguments"] is already a parsed dict
        result = run_tool(event["name"], event["arguments"])
        pending_tools.append({"call_id": event["call_id"], "result": result})
        await flush_if_idle()
    elif t in ("reply.started", "input.speech.started"):
        last_event = t
    elif t == "reply.done":
        last_event = t
        if event.get("status") == "interrupted":
            pending_tools.clear()
        else:
            await flush_if_idle()

The gotchas worth memorizing: event["arguments"] arrives as a parsed dict, but the result you send back must be a JSON-encoded string. Always echo the original call_id. For long-running tools, set execution_mode: "hold" so the agent stays quiet while you work, and use reply.create to slip in a status update.

Go multilingual

The Voice Agent API transcribes six input languages — English, French, German, Italian, Portuguese, and Spanish — and can speak those plus Hindi, Japanese, Korean, Mandarin, and Russian on the output side. (The agent can speak languages it can't yet transcribe.)

For voice selection: the US and UK voices (anna, george, vera) hold their character across languages, while native-accent voices — giovanni (Italian), lola (Spanish), juergen (German), rafael (Portuguese), estelle (French) — code-switch naturally between their primary language and English.

Worth noting: if you need broader language coverage for pure transcription, the standalone Universal-3.5 Pro Realtime streaming model supports 18 languages, and async Universal-2 covers 99+. The Voice Agent API's six-language input set is scoped to the full conversational loop.

Drop-in plugins: LiveKit and Pipecat

If you're already building on an orchestration framework, you don't have to hand-wire anything. LiveKit and Pipecat are partners, not competitors — we ship drop-in plugins for both so you can slot AssemblyAI's Voice AI in as your STT (or your full voice layer) with a few lines of config.

# Pipecat — use AssemblyAI streaming STT in your pipeline
# pip install "pipecat-ai[assemblyai]"
from pipecat.services.assemblyai import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
    # Universal-3.5 Pro Realtime under the hood
)
# Add `stt` to your Pipecat pipeline alongside your LLM and TTS services.

LiveKit works the same way — AssemblyAI plugs in as the STT node in your agent graph. Both frameworks handle transport and orchestration; AssemblyAI handles the accuracy. If your team has already standardized on one of them, this is the shortest path to swapping in a better speech-to-text layer.

Update your agent, then ship it

Here's the thing about voice agents: the model quality compounds. Every point of WER you shave off means fewer misheard names, fewer wrong tool calls, and fewer moments where the caller has to repeat themselves. That's not a cosmetic improvement — it's the difference between a demo and a system people actually trust with a phone number.

The Voice Agent API gives you that accuracy floor, natural turn-taking, and predictable pricing without asking you to become a distributed-systems expert first. Wire up one WebSocket, and the hard parts are handled.

Talk to a live agent built on the API — the fastest way to feel the turn-taking is to hear it. Try the live demo, then grab your API key and build your own.

Frequently asked questions

What is the AssemblyAI Voice Agent API and what does it actually handle for me?

It's a single WebSocket endpoint that runs the full voice-agent pipeline — speech-to-text, LLM reasoning, text-to-speech, neural turn detection, tool calling, and session resumption — for a flat $4.50/hr. You send audio and receive audio without wiring separate providers together.

How much does the Voice Agent API cost compared to alternatives like OpenAI's Realtime API?

It's a flat $4.50/hr, billed per second, covering STT, LLM, and TTS with no per-character or concurrency fees. That's roughly 4x cheaper than OpenAI's Realtime API, which runs around $18/hr once metered audio is factored in.

Which speech-to-text model powers the Voice Agent API and how accurate is it?

Universal-3.5 Pro Realtime, our streaming flagship. On the Pipecat voice-agent benchmark it posts 6.99% pooled WER and a 15.31% entity error rate — versus 50.5% entity error for Deepgram Flux — so names and phone numbers survive the call.

What's the WebSocket endpoint and how do I authenticate to it?

Connect to wss://agents.assemblyai.com/v1/ws (US) or wss://agents.eu.assemblyai.com/v1/ws (EU) with Authorization: Bearer YOUR_API_KEY. Browser clients should use a temporary token passed as ?token=... rather than shipping the key.

How does the Voice Agent API decide when a caller has finished speaking?

Neural end-of-turn detection reads tonality and pacing — not just silence — and decides in roughly 300ms. Unlike a VAD timer, it won't cut off a caller who pauses mid-sentence to think, which is why turn-taking feels natural.

Do I need a proprietary SDK, or can I use my existing framework?

No SDK required — it's standard JSON over WebSocket that works with any client in any language, and it plugs into coding agents like Claude Code. If you use LiveKit or Pipecat, drop-in plugins let you slot AssemblyAI in directly.