Agora voice agent with AssemblyAI Universal-3.5 Pro Realtime

Want to add speaker-aware, low-latency transcription to an Agora call without touching your client code? This guide shows you how to build an Agora voice agent where a Python server joins the channel as a silent observer, pulls raw PCM audio from every participant, and streams it straight to AssemblyAI's streaming speech-to-text WebSocket — powered by Universal-3.5 Pro Realtime, our streaming flagship.

Agora is your transport layer. AssemblyAI is your Voice AI layer. The two connect cleanly because Agora's Python Server SDK hands you raw PCM frames in exactly the format our streaming endpoint expects.

Agora + AssemblyAI: why pair them?

Agora handles the hard parts of real-time audio transport — global routing, jitter buffering, echo cancellation, WebRTC across browsers and mobile. What it doesn't do well is transcription. Its built-in STT is coarse, English-limited, and gives you no speaker awareness.

That's the gap AssemblyAI fills. Here's how Universal-3.5 Pro Realtime compares to Agora's built-in option:

That 6.99% WER is from the Pipecat open STT benchmark, which measures accuracy on real voice-agent conversations rather than clean read-aloud audio — the honest test for anything you're putting on a live call. For the full comparison against Deepgram Flux (15.58%), ElevenLabs Scribe v2 (9.76%), and Google Chirp3 (9.04%), see our benchmarks page.

Architecture

The data path is straightforward:

Browser/mobile clients → Agora channel → Python server bot → AssemblyAI Universal-3.5 Pro Realtime WebSocket → your application logic

The bot joins the Agora channel as a subscriber, receives PCM frames from every participant, and forwards them to AssemblyAI over a single WebSocket. Transcripts come back as Turn events at natural speech boundaries. Key specs:

• WebSocket endpoint: wss://streaming.assemblyai.com/v3/ws
• Audio format: PCM, 16-bit LE, 16 kHz sample rate, mono
• Frame spec: 160 samples per channel (10ms chunks) — streamed directly, no buffering

Quick start

git clone https://github.com/kelsey-aai/voice-agent-agora-universal-3-5-pro
cd agora-universal-3-5-pro
pip install agora-python-server-sdk websockets
cp .env.example .env
python bot.py --channel my-channel

The bot joins my-channel, begins streaming audio to AssemblyAI, and prints transcripts to stdout. Hit Ctrl+C to terminate the session cleanly.

Building along? Grab a free AssemblyAI API key — new accounts get free API credit to start, no credit card required.

Environment setup

# .env
AGORA_APP_ID=your_agora_app_id
AGORA_APP_CERT=your_agora_certificate
AGORA_CHANNEL=my-channel
AGORA_BOT_UID=9999
ASSEMBLYAI_API_KEY=your_assemblyai_api_key

Your Agora credentials come from the Agora Console; your AssemblyAI key comes from the dashboard.

Core integration

The connection to AssemblyAI is a single WebSocket URL with your parameters as query strings. Note the model ID — universal-3-5-pro — and that streaming auth uses a bare API key in the Authorization header (no Bearer prefix):

import os

SAMPLE_RATE = 16000
CHANNELS = 1  # mono

AAI_WS_URL = (
    "wss://streaming.assemblyai.com/v3/ws"
    f"?sample_rate={SAMPLE_RATE}"
    "&speech_model=universal-3-5-pro"
    "&format_turns=true"
)

HEADERS = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}


Here's the concurrent loop — pull audio frames from Agora, forward them to AssemblyAI, and handle transcript events as they arrive:
import asyncio, json, os, websockets

async def run_bot(agora_channel):
    async with websockets.connect(AAI_WS_URL, additional_headers=HEADERS) as ws:

        async def send_audio():
            async for frame in agora_channel.audio_frames():
                # frame.data is raw PCM16, 16 kHz mono — stream directly
                await ws.send(frame.data)
            await ws.send(json.dumps({"type": "Terminate"}))

        async def recv_loop():
            async for raw in ws:
                msg = json.loads(raw)
                if msg["type"] == "Begin":
                    print("Session started")
                elif msg["type"] == "Turn":
                    tag = "FINAL" if msg["end_of_turn"] else "partial"
                    print(tag, msg["transcript"])
                elif msg["type"] == "Termination":
                    return

        await asyncio.gather(send_audio(), recv_loop())

The streaming server sends four message types: Begin, SpeechStarted, Turn (with end_of_turn false for partials, true for finals), and Termination. Always send {"type":"Terminate"} when you're done so billing stops cleanly.

Audio format

Configure Agora to hand you unmixed, per-participant PCM at the right sample rate:

agora_channel.set_playback_audio_frame_before_mixing_parameters(
    num_of_channels=1,
    sample_rate=16000,
)
agora_channel.subscribe_all_audio()

Each frame.data is raw PCM, 16-bit LE, 16 kHz, mono, at 160 samples per channel (10ms). Universal-3.5 Pro Realtime accepts any chunk size between 50ms and 1000ms, so Agora's 10ms frames stream straight through without buffering.

Generating tokens for production

In production, mint an Agora token for the bot rather than running in test mode:

pip install agora-token-builder

Use RtcTokenBuilder.buildTokenWithUid() with the Role_Subscriber role (the bot only listens) and a sensible expiration — a 3600-second (1-hour) window is a reasonable default.

The differentiators worth knowing

A few things set Universal-3.5 Pro Realtime apart once you're past the plumbing:

• Neural end-of-turn detection. Instead of a fixed silence timer (VAD), the model reads tonality and pacing to decide when a speaker is genuinely done — landing in roughly 300ms. Callers who pause mid-sentence don't get cut off.
• agent_context. Passing conversational context to the model cuts WER by 10.2% on agent workloads — the transcript gets sharper as the conversation builds.
• Voice isolation. An optional add-on that strips background noise before transcription, which matters on the noisy mobile connections Agora often carries.

Pricing

Streaming with Universal-3.5 Pro Realtime is $0.45/hr base, billed per second, pay-as-you-go, with no minimums and unlimited concurrency. Full pricing is on the pricing page.

Building a full voice agent? Consider the Voice Agent API

This guide connects Agora's transport to AssemblyAI's standalone streaming STT — you bring your own LLM and TTS. That's the right call when you need full control over the pipeline.

But if what you actually want is a complete conversational agent — STT, LLM, TTS, and turn detection over one connection — the Voice Agent API is the shorter path. It's a flat $4.50/hr all-in (roughly 4x cheaper than OpenAI's Realtime API), and it handles the parts this guide leaves to you. Our Voice Agent API build guide walks through it end to end.

Want to hear it first? Talk to a live agent built on the Voice Agent API, then decide which path fits.

Frequently asked questions

How do I stream Agora channel audio to AssemblyAI for real-time transcription?

Run a Python server bot with the Agora Python Server SDK that joins the channel as a subscriber, pulls raw PCM16 frames, and forwards them to wss://streaming.assemblyai.com/v3/ws with speech_model=universal-3-5-pro. Transcripts return as Turn events.

Which speech-to-text model should I use for an Agora voice agent, and how accurate is it?

Universal-3.5 Pro Realtime, our streaming flagship. It posts a 6.99% pooled WER on the Pipecat voice-agent benchmark — versus 15.58% for Deepgram Flux — so names and numbers spoken on the call survive transcription.

How much does it cost to add AssemblyAI transcription to Agora?

Universal-3.5 Pro Realtime streaming is $0.45/hr base, billed per second with no minimums. If you want a full STT+LLM+TTS agent instead, the Voice Agent API is a flat $4.50/hr all-in.

How many languages does Universal-3.5 Pro Realtime support for streaming transcription?

The streaming flagship supports 18 languages. Note that the managed Voice Agent API supports 6 input languages for the full conversational loop, and async Universal-2 covers 99+ if you only need transcription.

Does the bot need to change my Agora client code?

No. The bot joins as a silent server-side subscriber and observes audio, so your existing browser and mobile clients are untouched. You add transcription entirely on the server.

Should I build directly on streaming STT or use the Voice Agent API?

Use standalone streaming STT when you want to bring your own LLM and TTS and control the whole pipeline. Use the Voice Agent API when you'd rather ship faster with STT, LLM, TTS, and turn detection handled over one WebSocket.