Now live New models for transcription and text-to-speech. AssemblyAI-STT-3.7 AssemblyAI-TTS-1
Insights & Use Cases

Agora voice agent with AssemblyAI-STT-3.7-Realtime-Agent

Add speaker-aware, low-latency transcription to an Agora call without touching your client code. Here's how a Python server bot streams raw PCM straight to AssemblyAI-STT-3.7-Realtime-Agent.

Abstract green mobius illustration

Written by

Kelsey Foster

Published on

18 November 2026

Want to add speaker-aware, low-latency transcription to an Agora call without touching your client code? This guide shows you how to build an Agora voice agent where a Python server joins the channel as a silent observer, pulls raw PCM audio from every participant, and streams it straight to AssemblyAI's streaming speech-to-text WebSocket — powered by AssemblyAI-STT-3.7-Realtime-Agent, our streaming model for voice agents.

Agora is your transport layer. AssemblyAI is your Voice AI layer. The two connect cleanly because Agora's Python Server SDK hands you raw PCM frames in exactly the format our streaming endpoint expects.

Agora + AssemblyAI: why pair them?

Agora handles the hard parts of real-time audio transport — global routing, jitter buffering, echo cancellation, WebRTC across browsers and mobile. What it doesn't do well is transcription. Its built-in STT is coarse, English-limited, and gives you no speaker awareness.

That's the gap AssemblyAI fills. Here's how AssemblyAI-STT-3.7-Realtime-Agent compares to Agora's built-in option:

Metric AssemblyAI-STT-3.7-Realtime-Agent Agora built-in STT
Word error rate (AssemblyAI voice-agent benchmark) 4.53% ~14–18%
Final transcript after the speaker stops (AssemblyAI research) 86 ms ~600–900ms
Neural end-of-turn detection ✅ semantic context + voice activity ❌ silence timer
Speaker diarization ✅ real-time ❌
LLM Gateway access ✅ ❌
Languages 32 (streaming) Limited
Audio formats PCM, μ-law, Opus PCM only

That 4.53% WER is from AssemblyAI's English voice-agent benchmark, the lowest word error rate of the 26 streaming models we ran. It measures accuracy on 12,460 voice-agent calls rather than clean read-aloud audio — the honest test for anything you're putting on a live call. On the same benchmark, Soniox Realtime v6 posts 6.02%, ElevenLabs Scribe v3 Live 7.38%, Deepgram Nova-3 8.64%, and Deepgram Flux EN 13.50%. On Pipecat's independent open STT benchmark, the previous model, Universal-3.6 Pro Realtime, posts a 0.96% pooled semantic WER and delivers the final transcript a median 307 ms after the speaker stops. For more, see our benchmarks page.

Architecture

The data path is straightforward:

Browser/mobile clients → Agora channel → Python server bot → AssemblyAI-STT-3.7-Realtime-Agent WebSocket → your application logic

The bot joins the Agora channel as a subscriber, receives PCM frames from every participant, and forwards them to AssemblyAI over a single WebSocket. Transcripts come back as Turn events at natural speech boundaries. Key specs:

  • WebSocket endpoint: wss://streaming.assemblyai.com/v3/ws
  • Audio format: PCM, 16-bit LE, 16 kHz sample rate, mono
  • Frame spec: 160 samples per channel (10ms chunks) — streamed directly, no buffering

Quick start

git clone https://github.com/kelsey-aai/voice-agent-agora-universal-3-pro
cd voice-agent-agora-universal-3-pro
pip install agora-python-server-sdk websockets
cp .env.example .env
python bot.py --channel my-channel

The bot joins my-channel, begins streaming audio to AssemblyAI, and prints transcripts to stdout. Hit Ctrl+C to terminate the session cleanly.

Building along? Grab a free AssemblyAI API key — new accounts get free API credit to start, no credit card required.

Environment setup

# .env
AGORA_APP_ID=your_agora_app_id
AGORA_APP_CERT=your_agora_certificate
AGORA_CHANNEL=my-channel
AGORA_BOT_UID=9999
ASSEMBLYAI_API_KEY=your_assemblyai_api_key

Your Agora credentials come from the Agora Console; your AssemblyAI key comes from the dashboard.

Core integration

The connection to AssemblyAI is a single WebSocket URL with your parameters as query strings. Note the model ID — assemblyai-stt-3-7-realtime-agent — and that streaming auth uses a bare API key in the Authorization header (no Bearer prefix):

import os

SAMPLE_RATE = 16000
CHANNELS = 1  # mono

AAI_WS_URL = (
    "wss://streaming.assemblyai.com/v3/ws"
    f"?sample_rate={SAMPLE_RATE}"
    "&speech_model=assemblyai-stt-3-7-realtime-agent"
)

HEADERS = {"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}


Here's the concurrent loop — pull audio frames from Agora, forward them to AssemblyAI, and handle transcript events as they arrive:
import asyncio, json, os, websockets

async def run_bot(agora_channel):
    async with websockets.connect(AAI_WS_URL, additional_headers=HEADERS) as ws:

        async def send_audio():
            async for frame in agora_channel.audio_frames():
                # frame.data is raw PCM16, 16 kHz mono — stream directly
                await ws.send(frame.data)
            await ws.send(json.dumps({"type": "Terminate"}))

        async def recv_loop():
            async for raw in ws:
                msg = json.loads(raw)
                if msg["type"] == "Begin":
                    print("Session started")
                elif msg["type"] == "Turn":
                    tag = "FINAL" if msg["end_of_turn"] else "partial"
                    print(tag, msg["transcript"])
                elif msg["type"] == "Termination":
                    return

        await asyncio.gather(send_audio(), recv_loop())

The streaming server sends four message types: Begin, SpeechStarted, Turn (with end_of_turn false for partials, true for finals), and Termination. Always send {"type":"Terminate"} when you're done so billing stops cleanly.

Stream Your First Agora Call

New accounts get free API credit—no credit card required. Grab a key and start streaming Agora channel audio to AssemblyAI-STT-3.7-Realtime-Agent in minutes.

Sign up free

Audio format

Configure Agora to hand you unmixed, per-participant PCM at the right sample rate:

agora_channel.set_playback_audio_frame_before_mixing_parameters(
    num_of_channels=1,
    sample_rate=16000,
)
agora_channel.subscribe_all_audio()

Each frame.data is raw PCM, 16-bit LE, 16 kHz, mono, at 160 samples per channel (10ms). AssemblyAI-STT-3.7-Realtime-Agent accepts any chunk size between 50ms and 1000ms, so Agora's 10ms frames stream straight through without buffering.

Generating tokens for production

In production, mint an Agora token for the bot rather than running in test mode:

pip install agora-token-builder

Use RtcTokenBuilder.buildTokenWithUid() with the Role_Subscriber role (the bot only listens) and a sensible expiration — a 3600-second (1-hour) window is a reasonable default.

The differentiators worth knowing

A few things set AssemblyAI-STT-3.7-Realtime-Agent apart once you're past the plumbing:

  • Neural end-of-turn detection. Instead of a fixed silence timer (VAD), the model combines semantic context with voice activity to decide when a speaker is genuinely done — the final transcript lands 86 ms after the speaker stops in AssemblyAI's published research. Callers who pause mid-sentence don't get cut off.
  • agent_context. Passing conversational context to the model makes the transcript sharper as the conversation builds.
  • Voice isolation. An optional add-on that strips background noise before transcription, which matters on the noisy mobile connections Agora often carries.
Build on the Streaming Flagship

Neural end-of-turn detection, agent_context, and voice isolation for noisy mobile calls. Get a free API key and put them to work on your Agora bot.

Sign up free

Pricing

Streaming with AssemblyAI-STT-3.7-Realtime-Agent is $0.45/hr base, billed per second, pay-as-you-go, with no minimums and no cap on open streaming sessions (only new sessions per minute are limited, and that limit scales with usage). Full pricing is on the pricing page.

Building a full voice agent? Add the voice output

This guide connects Agora's transport to AssemblyAI's standalone streaming STT — you bring your own LLM, and voice output is left to you. That's the right call when you need full control over the pipeline.

If what you actually want is a complete conversational agent, pair this bot with AssemblyAI's Text-to-Speech API for voice output and route each final Turn through your LLM — or let LiveKit or Pipecat run the loop with Agora still as the transport. Both speech ends then come from one vendor, and turn detection is already handled by AssemblyAI-STT-3.7-Realtime-Agent. The voice agent best practices walk through the pipeline end to end.

Ship the Full Agent Faster

Pair AssemblyAI-STT-3.7-Realtime-Agent with AssemblyAI's Text-to-Speech API and let LiveKit or Pipecat run the loop. The voice agent best practices take you from transcript to a working agent end to end.

Read the best practices

Frequently asked questions

How do I stream Agora channel audio to AssemblyAI for real-time transcription?

Run a Python server bot with the Agora Python Server SDK that joins the channel as a subscriber, pulls raw PCM16 frames, and forwards them to wss://streaming.assemblyai.com/v3/ws with speech_model=assemblyai-stt-3-7-realtime-agent. Transcripts return as Turn events.

Which speech-to-text model should I use for an Agora voice agent, and how accurate is it?

AssemblyAI-STT-3.7-Realtime-Agent, our streaming model for voice agents. It posts a 4.53% word error rate on AssemblyAI's English voice-agent benchmark, the lowest of the 26 streaming models we ran, versus 13.50% for Deepgram Flux EN — so names and numbers spoken on the call survive transcription.

How much does it cost to add AssemblyAI transcription to Agora?

AssemblyAI-STT-3.7-Realtime-Agent streaming is $0.45/hr base, billed per second with no minimums. If you want the agent to talk back, add AssemblyAI's Text-to-Speech API for voice output; current rates for both are on the pricing page.

How many languages does AssemblyAI-STT-3.7-Realtime-Agent support for streaming transcription?

The streaming flagship supports 32 languages with automatic language detection. Async Universal-2 covers 99+ if you only need pre-recorded transcription.

Does the bot need to change my Agora client code?

No. The bot joins as a silent server-side subscriber and observes audio, so your existing browser and mobile clients are untouched. You add transcription entirely on the server.

Should I build directly on streaming STT or use an orchestrator?

Build directly on streaming STT, as this guide does, when Agora is already your transport and you want to own the loop. Use LiveKit or Pipecat when you'd rather have the framework handle buffering, barge-in, and the handoffs between AssemblyAI-STT-3.7-Realtime-Agent, your LLM, and AssemblyAI's Text-to-Speech API.