LiveKit is how much of the voice-agent market ships—and AssemblyAI-STT-3.7-Realtime-Agent runs in its native AssemblyAI plugin, alongside AssemblyAI text-to-speech. Here's how to wire them together and avoid the one gotcha.
The model and endpoint
- Model: AssemblyAI-STT-3.7-Realtime-Agent, id assemblyai-stt-3-7-realtime-agent (the streaming parameter is speech_model, singular). Universal-3.6 Pro Realtime (universal-3-6-pro), the previous model, remains fully supported.
- LiveKit plugin: livekit-plugins-assemblyai 1.8.12+ is the first release that accepts assemblyai-stt-3-7-realtime-agent. The plugin still defaults to universal-3-6-pro, so pass model="assemblyai-stt-3-7-realtime-agent" explicitly. assemblyai.TTS ships in the same package from 1.8.9.
- Endpoint: wss://streaming.assemblyai.com/v3/ws. The old /v2/realtime/ws returns HTTP 410 — don't use it.
- Audio: PCM16 mono 16kHz; auth is your raw API key (no Bearer prefix on streaming STT).
- Pricing: $0.45/hr for AssemblyAI-STT-3.7-Realtime-Agent. For the speech-out side, AssemblyAI's Text-to-Speech API runs AssemblyAI-TTS-1 in the same LiveKit pipeline at $15 per 1M characters.
Streaming setup in Python
The core streaming loop looks like this — it's the same speech layer the LiveKit plugin runs under the hood:
# pip install "assemblyai>=1.10.1"
import os
from assemblyai.streaming.v3 import (
RealTimeEvents, RealTimeParameters, RealTimeSessionParameters,
RealTimeTranscriber, RealTimeTranscriberOptions, TurnEvent,
)
def on_turn(_, event: TurnEvent):
tag = "FINAL" if event.end_of_turn else "partial"
print(f"{tag}: {event.transcript}")
client = RealTimeTranscriber(
RealTimeTranscriberOptions(),
api_key=os.environ["ASSEMBLYAI_API_KEY"],
)
client.on(RealTimeEvents.Turn, on_turn)
client.connect(RealTimeParameters(
sample_rate=16000,
speech_model="assemblyai-stt-3-7-realtime-agent",
agent_context="What's your email address?",
))
# feed 16kHz mono PCM16 chunks (50-1000ms) via client.stream(chunk)
# after each agent reply, update the context:
client.set_params(RealTimeSessionParameters(agent_context="Sure, what date would you like to book?"))
client.disconnect(terminate=True) # ALWAYS terminateAdd the voice: a full AgentSession with AssemblyAI-TTS-1
The same livekit-plugins-assemblyai package also ships assemblyai.TTS, which speaks your agent's replies with AssemblyAI-TTS-1 (assemblyai-tts-1), so one AssemblyAI key covers both speech ends. Here's a complete AgentSession with AssemblyAI-STT-3.7-Realtime-Agent for speech-to-text and AssemblyAI-TTS-1 for the voice:
# pip install "livekit-agents[openai,silero]~=1.8" "livekit-plugins-assemblyai>=1.8.12"
from livekit import agents
from livekit.agents import Agent, AgentSession, TurnHandlingOptions
from livekit.plugins import assemblyai, openai, silero
VAD_THRESHOLD = 0.2 # AssemblyAI's default; keep Silero in step with it
async def entrypoint(ctx: agents.JobContext):
await ctx.connect()
tts = assemblyai.TTS(model="assemblyai-tts-1", voice="aai-kyla")
tts.prewarm() # open the TTS session before the first reply
session = AgentSession(
stt=assemblyai.STT(
model="assemblyai-stt-3-7-realtime-agent",
mode="balanced",
vad_threshold=VAD_THRESHOLD,
),
llm=openai.LLM(model="gpt-4o"),
tts=tts,
vad=silero.VAD.load(activation_threshold=VAD_THRESHOLD),
turn_handling=TurnHandlingOptions(
turn_detection="stt",
endpointing={"min_delay": 0},
),
use_tts_aligned_transcript=True,
)
await session.start(room=ctx.room, agent=Agent(instructions="You are a helpful voice assistant."))
if __name__ == "__main__":
agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))Pass voice="aai-kyla" explicitly: the plugin has its own default voice, which differs from the API's. tts.prewarm() opens the TTS session while the agent joins the room, so the first reply doesn't wait on a handshake, and use_tts_aligned_transcript=True keeps only the words the user actually heard when they interrupt. AssemblyAI-TTS-1 is $15 per 1M characters and speaks six languages. The Streaming TTS on LiveKit guide covers voices, languages and the EU data zone.
AssemblyAI-STT-3.7-Realtime-Agent is a native streaming STT plugin in LiveKit Agents. Get a free API key and wire it into your agent in minutes.
agent_context / Context Carryover
AssemblyAI-STT-3.7-Realtime-Agent takes an agent_context parameter — you feed it what your agent just said and it biases recognition toward the expected reply. On Universal-3.6 Pro Realtime, across 20,000 voice-agent files, this cut WER by 10.2%. Rolling context is on by default (prior finalized user turns carry over automatically), and LiveKit supports it out of the box, so you get it without predefining key terms.
The turn-detection gotcha (read this)
If you use turn_detection="stt", set mode="balanced" and endpointing={"min_delay": 0}. STT-based turn detection is the recommended setup: in balanced mode, AssemblyAI checks for the end of the turn after 128 ms of silence, ends it when the transcript reads as a finished sentence, holds it open while a number or other entity is still in progress, and forces it after 1,280 ms. But with mode unset, the LiveKit plugin sends 100 ms for both thresholds, and LiveKit adds up to 500 ms of its own endpointing delay on top. Setting the mode, and setting endpointing={"min_delay": 0} in TurnHandlingOptions on your AgentSession, fixes both. This is the single most common way a LiveKit + AssemblyAI setup goes wrong, so wire it up correctly from the start.
If you'd rather use LiveKit's turn-detector model, use MultilingualModel(), keep the plugin defaults (min_turn_silence=100, max_turn_silence=100), set endpointing={"min_delay": 0.5, "max_delay": 3.0}, and keep Silero VAD, which that mode requires. The LiveKit guide has the full settings for both.
How accurate is it?
On AssemblyAI's English voice-agent benchmark, AssemblyAI-STT-3.7-Realtime-Agent has the lowest normalized WER of the 26 streaming models we ran, 4.53%, and the lowest entity error rate, 13.2%, ahead of Soniox Realtime v6, ElevenLabs Scribe v3 Live and Deepgram Nova-3 (November 2026). Against Universal-3.6 Pro Realtime, it makes 80% fewer number errors on Voice Code Bench and delivers the final transcript 86 ms after speech ends, against 172 ms. Full numbers are in the AssemblyAI-STT-3.7-Realtime-Agent research post.
Diarization and deployment
AssemblyAI-STT-3.7-Realtime-Agent supports realtime speaker labels — useful for multi-party rooms. For deployment, the plugin runs anywhere your LiveKit agent runs, on LiveKit Cloud or your own infrastructure. The Universal-3.6 Pro on LiveKit guide covers setup, recommended turn-detection settings, and the self-hosted interruption filters.
Build it
Talk to a live agent to hear it in action, or get your free API key and drop AssemblyAI-STT-3.7-Realtime-Agent into your LiveKit agent.
AssemblyAI-STT-3.7-Realtime-Agent at $0.45/hr, with AssemblyAI-TTS-1 for the voice-output side. Grab a free API key and get Context Carryover out of the box.
Frequently asked questions
Does AssemblyAI work with LiveKit out of the box?
Yes. AssemblyAI is a native streaming STT plugin in LiveKit Agents, running AssemblyAI-STT-3.7-Realtime-Agent when you pass model="assemblyai-stt-3-7-realtime-agent" (livekit-plugins-assemblyai 1.8.12+), and it supports Context Carryover out of the box — no custom integration required. The same package ships assemblyai.TTS for AssemblyAI-TTS-1.
Which turn-detection setting should I use?
STT-based turn detection is the recommended setup: set turn_detection="stt" with mode="balanced" (min_turn_silence=128, max_turn_silence=1280) and endpointing={"min_delay": 0}. If you prefer LiveKit's turn-detector model, use MultilingualModel() with the plugin defaults and keep Silero VAD.
Which model and endpoint does the plugin use?
Pass model="assemblyai-stt-3-7-realtime-agent" to run AssemblyAI-STT-3.7-Realtime-Agent over wss://streaming.assemblyai.com/v3/ws; the plugin's default is still universal-3-6-pro. It needs livekit-plugins-assemblyai 1.8.12 or later, and it's $0.45/hr.
How do I pass conversation context?
Use agent_context: seed it at connection time and update it after each agent reply. On Universal-3.6 Pro Realtime it cut WER by 10.2% across 20,000 voice-agent files, and prior finalized user turns carry over automatically.
How accurate is it for voice agents?
On AssemblyAI's English voice-agent benchmark, AssemblyAI-STT-3.7-Realtime-Agent has the lowest normalized WER (4.53%) and the lowest entity error rate (13.2%) of the 26 streaming models we ran (November 2026).