Daily.co voice agent with AssemblyAI Universal-3 Pro Streaming

This is the bare-metal guide to building a Daily.co voice agent — WebRTC transport from Daily.co, speech-to-text from AssemblyAI's Universal-3.5 Pro Realtime, reasoning from an LLM, and speech from a TTS engine — wired together directly, without Pipecat. If you want to see exactly how Daily's audio tracks connect to AssemblyAI's WebSocket, this shows you every seam.

Daily.co is your transport partner. AssemblyAI is your Voice AI layer. The daily-python SDK gives you raw PCM frames; AssemblyAI's streaming speech-to-text endpoint consumes them directly.

Architecture: the Daily.co + AssemblyAI pipeline

The full loop runs like this:

Browser/phone → Daily.co WebRTC → daily-python SDK → AssemblyAI WebSocket → OpenAI GPT-4o → Cartesia TTS → PCM audio back to the Daily room

Daily handles global audio routing and WebRTC. AssemblyAI turns the audio into text. GPT-4o decides what to say. Cartesia turns that back into speech, which Daily plays into the room. Four services, one clean data path.

Why Universal-3.5 Pro Realtime for a Daily voice agent?

On a live call, transcription accuracy is the whole game — if the STT layer mishears a name or a phone number, no amount of clever prompting downstream saves the interaction. Universal-3.5 Pro Realtime posts a 6.99% pooled WER on the Pipecat open STT benchmark, which measures real voice-agent conversations rather than clean read-aloud audio:

The entity error rate is what decides whether your Daily agent captures the callback number correctly. At 15.31% versus Deepgram Flux's 50.5%, that's not a rounding difference — it's whether the agent works. See the full methodology on our benchmarks page.

Prerequisites

• Python 3.11+
• An AssemblyAI API key
• A Daily.co API key
• An OpenAI API key
• A Cartesia API key

Getting started? Grab a free AssemblyAI API key — new accounts get free API credit to start, no credit card required.

AssemblyAI connection parameters

The connection is a single WebSocket. Note the model ID — universal-3-5-pro — and that streaming auth uses a bare API key in the Authorization header (no Bearer prefix):

import os

SAMPLE_RATE = 16000  # pcm_s16le, 16 kHz mono

AAI_WS_URL = (
    "wss://streaming.assemblyai.com/v3/ws"
    f"?sample_rate={SAMPLE_RATE}"
    "&speech_model=universal-3-5-pro"
    "&format_turns=true"
)

HEADERS = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}

‍

How audio flows

The daily-python SDK fires a callback for each participant's audio frames. Forward that PCM straight to the AssemblyAI WebSocket:

import asyncio

def on_audio_data(self, participant_id, audio_data, sample_rate, num_channels):
    if self.aai_ws and not self.aai_ws.closed:
        # audio_data is raw PCM16 — stream directly, no buffering
        asyncio.create_task(self.aai_ws.send(audio_data))

‍

On the receive side, read transcripts as Turn events, pass finals to GPT-4o, and route the reply text to Cartesia:

import json

async def recv_transcripts(ws, on_final):
    async for raw in ws:
        msg = json.loads(raw)
        if msg["type"] == "Turn" and msg["end_of_turn"]:
            await on_final(msg["transcript"])  # -> GPT-4o -> Cartesia -> Daily
        elif msg["type"] == "Termination":
            return

The streaming server sends Begin, SpeechStarted, Turn (with end_of_turn false for partials, true for finals), and Termination. Always send {"type":"Terminate"} when the call ends so billing stops cleanly.

Why direct Daily.co instead of Pipecat?

Pipecat is a great orchestration framework — and AssemblyAI ships a drop-in Pipecat plugin, so it's a first-class path. But building the Daily integration by hand is worth doing at least once. You see exactly where audio enters, how turn events map to LLM calls, and where latency actually accumulates. When something breaks in production, you'll know which seam to check because you wired every one of them yourself.

If you'd rather not hand-wire it, the Pipecat route gets you the same Universal-3.5 Pro Realtime accuracy with far less code — see our Pipecat and Agora integration guides for the framework-first approach.

The differentiators worth knowing

• Neural end-of-turn detection. Rather than a fixed silence timer, the model reads tonality and pacing to decide when a speaker is genuinely done — in roughly 300ms. Callers who pause mid-sentence don't get interrupted.
• agent_context. Feeding conversational context to the model cuts WER by 10.2% on agent workloads.
• Fewer hallucinations, better entity capture. Phone numbers, emails, and names — the things a caller actually needs the agent to get right — come through cleanly.

Pricing

Streaming with Universal-3.5 Pro Realtime is $0.45/hr base, billed per second, pay-as-you-go, with no minimums and unlimited concurrency. (You pay Daily, OpenAI, and Cartesia separately in this bring-your-own stack.) Full pricing is on the pricing page.

Why not just use the Voice Agent API?

Fair question — and for a lot of teams, the answer is "you should."

This guide wires four vendors together: Daily for transport, AssemblyAI for STT, OpenAI for reasoning, Cartesia for speech. That's the right architecture when you need specific control over each piece. But it's also four bills, four integrations, and four places for latency to creep in.

The Voice Agent API collapses STT, LLM, TTS, and turn detection into one WebSocket at a flat $4.50/hr all-in — roughly 4x cheaper than OpenAI's Realtime API (~$18/hr) and far less to maintain. AssemblyAI is invisible infrastructure: we're not the agent, we're the platform you build it on. If you don't specifically need the bring-your-own control this guide gives you, start there. The Voice Agent API build guide covers it end to end.

Hear it before you build it. Talk to a live agent running on the Voice Agent API, then pick the path that fits.

Frequently asked questions

How do I stream Daily.co audio to AssemblyAI for real-time transcription?

Use the daily-python SDK's audio callback to grab raw PCM16 frames per participant and forward them to wss://streaming.assemblyai.com/v3/ws with speech_model=universal-3-5-pro. Read transcripts back as Turn events and route finals to your LLM.

Should I build a Daily voice agent directly or use Pipecat?

Build it directly when you want to understand and control every seam — audio in, turn events, LLM calls, TTS out. Use Pipecat's drop-in AssemblyAI plugin when you'd rather ship faster with the same Universal-3.5 Pro Realtime accuracy and less code.

What latency can I expect from Daily.co plus Universal-3.5 Pro Realtime?

AssemblyAI's streaming layer adds low hundreds of milliseconds, with neural end-of-turn detection landing in roughly 300ms. Total round-trip latency also depends on Daily's transport and your chosen LLM and TTS, so measure the full pipeline end to end.

How much does it cost to run a Daily.co voice agent on AssemblyAI?

Universal-3.5 Pro Realtime streaming is $0.45/hr base, billed per second, with Daily, OpenAI, and Cartesia billed separately. A fully managed alternative is the Voice Agent API at a flat $4.50/hr all-in.

How many languages does Universal-3.5 Pro Realtime support?

The streaming flagship supports 18 languages. The managed Voice Agent API supports 6 input languages for the full conversational loop, and async Universal-2 covers 99+ if you only need transcription.

Why not just use the AssemblyAI Voice Agent API instead of wiring four vendors?

If you don't need bring-your-own control over each component, you probably should. The Voice Agent API bundles STT, LLM, TTS, and turn detection into one WebSocket at $4.50/hr — fewer bills, fewer integrations, and less latency between seams.