Updated November 2026 for AssemblyAI-STT-3.7-Realtime-Agent. Benchmark figures come from AssemblyAI's published research and our benchmarks page. For a ranked vendor list and streaming prices, see the best real-time speech-to-text APIs.
If you're building a voice agent, the speech-to-text API underneath it decides whether the conversation feels natural or broken. In our voice agent report, 76% of builders rated speech-to-text accuracy as critical, and 95% of respondents said they've been frustrated with a voice agent at some point. The top frustration was having to repeat themselves.
Leaderboards won't tell you which API stops that from happening on your calls. A test plan will. This guide gives you seven tests that predict how a speech layer behaves inside a live agent, what a good result looks like, and how to run each one on your own audio. If you're choosing a speech-to-text API for something other than an agent, such as batch analytics or captioning, our general framework for choosing a speech-to-text API is the better starting point.
What is the best speech-to-text API for voice agents in 2026?
The best speech-to-text API for voice agents is one built for live conversation: accurate on names, emails and numbers, quick to deliver a final transcript, able to tell a pause from a finished thought, and able to ignore voices that aren't talking to the agent.
AssemblyAI-STT-3.7-Realtime-Agent is built for that job. Of the 26 streaming models we ran on AssemblyAI's voice-agent benchmark (12,460 English calls, November 2026), it has the lowest normalized word error rate (4.53%, ahead of Soniox Realtime v6 at 6.02%) and the lowest entity error rate (13.2%, ahead of Soniox Realtime v6, ElevenLabs Scribe v3 Live and Deepgram Nova-3). It costs $0.45/hr. But the right answer for your agent depends on your latency budget, your callers and your audio, so run the seven tests below before you commit.
Realtime or Realtime Agent? A simple rule
AssemblyAI has two current streaming models, and the choice comes down to one question: does software respond to the person speaking?
If it does, as in a voice agent that answers a caller, use AssemblyAI-STT-3.7-Realtime-Agent. If people read the transcript, live or later, use AssemblyAI-STT-3.7-Realtime.
| AssemblyAI-STT-3.7-Realtime-Agent | AssemblyAI-STT-3.7-Realtime | |
|---|---|---|
| Use it when | Software answers the speaker: phone agents, web voice agents, voice assistants | People read the transcript: notetakers, live captions, call notes, agent assist |
speech_model |
assemblyai-stt-3-7-realtime-agent |
assemblyai-stt-3-7-realtime |
| Price | $0.45/hr | $0.30/hr |
| Turn detection | Tuned to drive turn-taking; first partial as soon as speech starts by default | Tuned to segment a transcript; no end-of-turn check before 320 ms of silence |
| Agent-only controls | Voice Focus, manual turn mode with ForceEndpoint, interruption_delay |
None of these |
| Shared | 32 languages, keyterm prompting, context carryover with agent_context, real-time speaker labels, same WebSocket endpoint and SDK |
|
Two edge cases. If a human agent is on the call and your software only suggests what to say, nothing answers the caller automatically, so the Realtime model fits. If you need speaker labels on a recording of both sides of a call, the docs recommend the Realtime model for diarization. Universal-3.6 Pro Realtime is the previous model and is still fully supported for integrations pinned to it.
The rest of this guide is about agents, so every test and example assumes the Agent model.
Why voice agents need different tests
In batch transcription, speed is a convenience. In a voice agent, the speech layer is inside a loop: the caller speaks, the speech-to-text API finalizes a turn, an LLM decides what to say, and text-to-speech says it. Every error and every millisecond in the first step carries through the rest.
Real conversations are messy, too: interruptions, corrections, thinking pauses, a TV in the background, and a lot of "yes", "no" and "mm-hmm". An API tuned on recorded podcasts won't hold up in a live exchange, and average word error rate won't warn you.
So test the behaviors that break agents, not the averages that look good on a slide.
Test 1: Measure the whole latency budget, not processing time
What to test: the time from the moment the caller stops speaking to the moment your agent starts talking back. Then break out how much of it the speech layer takes.
People expect quick replies. Conversation research finds typical gaps between turns of only a few hundred milliseconds, so an agent that takes much longer feels robotic. The speech layer shares that budget with the LLM and text-to-speech.
What good looks like: a vendor that publishes end-of-speech-to-final numbers and says what they include. On AssemblyAI's latency benchmark, which gives each model the end of turn and times what's left, AssemblyAI-STT-3.7-Realtime-Agent returns the final transcript 86 ms after the end of speech, against 172 ms for Universal-3.6 Pro Realtime. That's delivery time; it doesn't include how long the model waits before deciding the turn is over. On the open pipecat-ai/stt-benchmark, which includes endpointing, the previous model, universal-3-6-pro, posts a 307 ms median time to final transcript, faster than AWS Transcribe, Azure, Google, Speechmatics and OpenAI on that table; Pipecat hasn't published a 3.7 result yet.
Be clear about the trade-off between speed and accuracy. A fast model that mishears the account number forces a repeat, and a repeat costs seconds. Our real-time STT latency benchmarks explain which latency each benchmark measures and how to time each stage.
Test 2: Score entity accuracy by class
What to test: accuracy on the specific tokens your agent acts on, split by type. When your agent hears john.smith@company.com as johnsmith@company.calm, you've lost the customer, and a 5% word error rate won't tell you that happened.
Entity classes fail differently. Here's AssemblyAI's voice-agent benchmark (12,460 scripted scenarios, 522 speakers, telephony audio with no, low or heavy background noise, November 2026) for four streaming systems agents commonly shortlist. Lower is better.
| Entity error rate | AssemblyAI-STT-3.7-Realtime-Agent | Soniox Realtime v6 | ElevenLabs Scribe v3 Live | Deepgram Nova-3 |
|---|---|---|---|---|
| All entities | 13.2% | 15.2% | 16.9% | 25.1% |
| Phone numbers | 2.03% | 2.20% | 2.96% | 4.35% |
| Email addresses | 13.98% | 15.24% | 16.26% | 31.46% |
| Names | 10.18% | 12.47% | 14.02% | 23.69% |
| Dates and times | 17.66% | 26.22% | 30.09% | 43.89% |
| Numbers and currency | 12.38% | 19.45% | 16.88% | 16.70% |
Three things to take from this. Overall entity error is lowest on the Agent model of the 26 streaming models we ran, ahead of Soniox Realtime v6, ElevenLabs Scribe v3 Live and Deepgram Nova-3, and it makes fewer errors than every other provider on each entity class shown, including phone numbers. Classes still behave differently, so if your agent books deliveries or collects account codes, add street addresses and codes to your own test set. And the spread between providers is wide: Deepgram Nova-3 gets 31.46% of email addresses wrong.
Dense numeric audio is its own test. On Voice Code Bench, 300 recordings packed with numbers, codes and identifiers, the Agent model's number error rate is 2.27% against 11.28% for Universal-3.6 Pro Realtime, a reduction of 80%. Money amounts with an error drop from 13.3% to 0.0%, strings of digits from 23.6% to 3.6%, and recordings with at least one missed entity from 71.3% to 37.0%. The STT-3.7-Realtime-Agent research post has every table and the method.
How to run it: pull 50 to 100 real calls, mark every entity in a reference transcript, and score each class separately. For the reasoning behind entity-first scoring, see why word error rate is broken.
Stream a call or talk live in the playground, and check how names, numbers and account IDs come through before you run the full test plan.
Test 3: Stress turn detection with hesitations
What to test: whether the speech layer waits through a natural pause mid-thought and ends the turn promptly when the caller is done.
Basic voice activity detection treats every pause as the end of a turn. But many speech segments after a pause are the same speaker continuing. Picture someone saying "My email is... john.smith@company.com" and the agent jumping in before they finish.
What good looks like: turn detection that reads what's been said, not only how long the silence is. AssemblyAI-STT-3.7-Realtime-Agent ends a turn when the transcript reads as a finished sentence, and holds it open when the turn ends in a number, ID or other entity that may still continue. On AssemblyAI's voice-agent benchmark, its end-of-turn precision is 78.97%, up 2.9 points from Universal-3.6 Pro Realtime, which means 12% fewer false end-of-turn events and fewer callers cut off mid-thought. Entity-aware endpointing is on by default in every mode, and you can adjust min_turn_silence and max_turn_silence mid-stream. If you'd rather decide turn ends yourself, with your own VAD, push-to-talk or a turn model in your orchestrator, connect with mode set to manual and send ForceEndpoint; that mode is Agent-only. The turn detection docs cover every parameter.
How to run it: script 20 turns with hesitations, self-corrections and dictated numbers. Count early cutoffs and late finals.
Test 4: Check how the model uses conversation context
What to test: whether the transcript of a short answer improves when the model knows what the agent just asked. "Two" after "How many tickets?" and "to" after "Where are you headed?" sound the same.
What good looks like: context that carries forward without extra plumbing. The STT 3.7 models carry the last five finalized turns forward automatically, up to 1,750 characters. You pass your agent's spoken reply as agent_context after each response, and the model uses it for the caller's next turn. Context is per session, so a reconnect starts fresh.
How to run it: run the same set of short answers with and without agent_context, and compare entity and short-reply errors.
Test 5: Time the barge-in signal
What to test: how quickly your agent learns that the caller has started talking over it, and whether background noise triggers false interruptions.
A speech-to-text API doesn't stop your agent from talking. It tells you when to, and your agent does the stopping.
What good looks like: a clear, early signal that ignores noise. The streaming API sends a SpeechStarted message once per turn, just before the first Turn message, and only once the model produces an actual transcript, so noise alone doesn't fire it. On the Agent model, interruption_delay defaults to 0, so the first partial follows immediately. If your agent's own TTS audio or a noisy room triggers false barge-ins, raise vad_threshold or interruption_delay, or turn on Voice Focus.
How to run it: play agent audio, have a tester interrupt at random points, and measure the gap from first syllable to playback stop. Then run the same test with a TV on in the background and count false stops.
Test 6: Test short replies and background voices together
What to test: two failures that trade off against each other. One is dropped short replies: a quiet "yes" that comes back as nothing. The other is background-speech leakage: words from a TV or a coworker transcribed as the caller's. Suppress background voices harder and you usually lose more quiet replies, so score them as a pair.
What good looks like: published numbers on both sides. On AssemblyAI's voice-agent benchmark, AssemblyAI-STT-3.7-Realtime-Agent drops 38% fewer short replies than Universal-3.6 Pro Realtime (0.18% vs 0.29%), gets 29% fewer short replies wrong, and adds about half as many words while the caller is silent (1.34 vs 2.53 per 1,000 words). Against the other providers we measured on the same calls, it drops the fewest short replies, ahead of Soniox Realtime v6, ElevenLabs Scribe v3 Live and Deepgram Nova-3, and no other provider has both less background speech and fewer dropped short replies. For noisy rooms, Voice Focus (voice_focus, near-field or far-field) isolates the primary speaker before audio reaches the model.
How to run it: include one- and two-word answers, quiet speakers and recordings with real background voices in your test set, and score them separately from clean audio.
Test 7: Match the transport, then test resilience
What to test: whether the API offers the right transport for each part of your pipeline, and how your agent recovers when a connection drops mid-call.
There are three ways to get speech into text, and an agent can use more than one:
- Streaming (WebSocket): continuous transcription of an open microphone, with turn detection on the server. The default for phone and web agents.
- Sync (one HTTP request): if you already detect turns yourself with a VAD or your orchestrator, send each finished utterance and get a transcript back in the response.
- Async (submit and poll): long recordings you'll analyze after the call.
AssemblyAI's Sync API returns a Universal-3.6 Pro transcript in about 134 ms at the median, at $0.30/hr, with no WebSocket and no polling:
curl -X POST https://sync.assemblyai.com/v1/transcribe \
-H "Authorization: $ASSEMBLYAI_API_KEY" \
-H 'X-AAI-Model: universal-3-6-pro' \
-F 'audio=@turn.wav;type=audio/wav'It fits push-to-talk and post-turn-detection transcription. You can pass prior turns as conversation_context to keep names consistent across a call.
How to run it: for streaming, kill the network mid-call and time the reconnect, then confirm your agent re-sends agent_context and keyterms on the new session. Billing, session limits and rate limits differ by vendor and change your bill at scale, so price on session time, not only audio time.
Evaluate on your own calls: a checklist
Benchmarks narrow the field. Your audio decides it. Before you sign anything, run your top two or three APIs through this checklist on their free tiers:
- Collect 50 to 100 real calls with your accents, noise and line quality, including the hard ones.
- Write reference transcripts by hand, and tag every entity by class.
- Score entity error by class, not just overall WER (Test 2).
- Script 20 hesitation turns and count early cutoffs (Test 3).
- Run short answers with and without context (Test 4).
- Time barge-in, with and without background noise (Test 5).
- Score dropped short replies and background-speech leakage as a pair (Test 6).
- Measure the full loop from end of speech to first agent audio, at your expected concurrency (Test 1).
- Break a connection mid-call and check recovery (Test 7).
- Price it on session time, not just audio time, including add-ons.
Get an API key and stream AssemblyAI-STT-3.7-Realtime-Agent into your agent, with keyterms and context carryover built in and no spend commitment.
A voice agent in practice
With AssemblyAI-STT-3.7-Realtime-Agent, you open one WebSocket, stream audio, and act on events. Here's the speech side of an agent in Python with the AssemblyAI SDK (1.10.1 or later):
import os
from assemblyai.streaming.v3 import (
RealTimeEvents,
RealTimeParameters,
RealTimeSessionParameters,
RealTimeTranscriber,
RealTimeTranscriberOptions,
SpeechStartedEvent,
TurnEvent,
)
GREETING = "Thanks for calling. Can I get the order number?"
def generate_reply(user_text: str) -> str:
# Call your LLM here, then speak the reply with TTS.
return "Got it. And what's the email on the order?"
def on_speech_started(client: RealTimeTranscriber, event: SpeechStartedEvent) -> None:
print("Caller started speaking: stop TTS playback")
def on_turn(client: RealTimeTranscriber, event: TurnEvent) -> None:
if event.end_of_turn and event.transcript:
reply = generate_reply(event.transcript)
# Tell the model what the agent just said, for the caller's next turn.
client.set_params(RealTimeSessionParameters(agent_context=reply))
client = RealTimeTranscriber(
RealTimeTranscriberOptions(),
api_key=os.environ["ASSEMBLYAI_API_KEY"],
)
client.on(RealTimeEvents.SpeechStarted, on_speech_started)
client.on(RealTimeEvents.Turn, on_turn)
client.connect(
RealTimeParameters(
speech_model="assemblyai-stt-3-7-realtime-agent",
sample_rate=16000,
agent_context=GREETING,
keyterms_prompt=["order number", "tracking ID"],
)
)
# Stream 16 kHz, 16-bit mono PCM with client.stream(...), and always finish with
# client.disconnect(terminate=True) so the session stops billing.Each Turn event carries the transcript so far and an end_of_turn flag. When it flips to true, hand the text to your LLM and stream the reply through text-to-speech. Keyterms and agent_context can be updated mid-conversation without reconnecting. The voice agent best practices guide in the docs walks through the full loop. The full script, plus a Sync API version for agents that detect turns themselves, is in the companion repo. Use it as a harness for Tests 4, 5 and 7.
Build the full voice agent
Speech-to-text is the ears. A working agent also needs a brain and a voice, and AssemblyAI covers all three on one API key:
- Listen: AssemblyAI-STT-3.7-Realtime-Agent, streaming speech-to-text with entity-aware endpointing, $0.45/hr.
- Think: LLM Gateway, an OpenAI-compatible endpoint that routes to models from multiple providers with your AssemblyAI key.
- Speak: AssemblyAI-TTS-1, streaming text-to-speech built for voice agents, trained on millions of hours of audio, including real conversational data. It costs $15 per 1M characters.
Wire them together with LiveKit, Pipecat or your own loop. LiveKit's AssemblyAI plugin already supports the Agent model in its latest release; for other frameworks, check their release notes for the 3.7 model IDs. For how the layers fit, see the Voice AI stack for building agents.
Conclusion
The gap between a voice agent that feels natural and one that feels broken usually comes down to a handful of speech-layer behaviors: a dropped digit, a turn cut short, a quiet "yes" that never arrives, a TV line acted on as if the caller said it. None of them show up in an average word error rate. All of them show up in the seven tests above.
So shortlist on published benchmarks, pick the right model for the job with the Realtime-vs-Agent rule, then let your own calls make the decision. The API that gets your callers' names, numbers and turn ends right is the one to ship.
Sign up free and connect AssemblyAI-STT-3.7-Realtime-Agent, LLM Gateway and AssemblyAI-TTS-1 on one API key.
Frequently Asked Questions (FAQs)
What is the best speech-to-text API for voice agents?
The best speech-to-text API for a voice agent is the one that gets entities, short replies and turn ends right on your own calls. On AssemblyAI's voice-agent benchmark, AssemblyAI-STT-3.7-Realtime-Agent has the lowest normalized word error rate and entity error rate of the 26 streaming models we ran, ahead of Soniox, ElevenLabs and Deepgram, at $0.45/hr. Shortlist on benchmarks like that, then run the seven tests in this guide on your own audio.
Should I use AssemblyAI-STT-3.7-Realtime or STT-3.7-Realtime-Agent?
Use AssemblyAI-STT-3.7-Realtime-Agent ($0.45/hr) when software responds to the person speaking, such as a phone or web voice agent. Use AssemblyAI-STT-3.7-Realtime ($0.30/hr) when people read the transcript, such as notetakers, live captions, call notes and agent assist. The Agent model adds Voice Focus, a manual turn mode and turn detection tuned for turn-taking.
How do I test speech-to-text entity accuracy for a voice agent?
Collect 50 to 100 real calls, write reference transcripts, tag every entity by class (names, phone numbers, emails, codes, addresses), and score each class separately. Overall word error rate hides the misses that break agent tasks. Results vary a lot by class: on AssemblyAI's voice-agent benchmark, error rates on email addresses range from 13.98% to 31.46% across the four systems compared above.
What is agent_context in AssemblyAI streaming?
agent_context is a streaming parameter that passes your voice agent's last spoken reply to the model, so it can use it when transcribing the caller's next turn. You set it at connection time and update it after each reply without reconnecting. The model also carries the last five finalized caller turns forward automatically, up to 1,750 characters.
Should a voice agent use streaming or a synchronous speech-to-text API?
Use streaming when you need continuous transcription of an open microphone with server-side turn detection, which covers most phone and web agents. Use a synchronous API when you already detect turns yourself and want to send finished utterances, such as push-to-talk. AssemblyAI's Sync API returns a Universal-3.6 Pro transcript in about 134 ms at the median over one HTTP request, at $0.30/hr.
How do I handle barge-in with a streaming speech-to-text API?
Listen for the speech layer's start-of-speech signal and stop text-to-speech playback when it arrives. AssemblyAI's streaming API sends a SpeechStarted message only once the model produces an actual transcript, so background noise alone doesn't trigger it. If your agent's own audio causes false stops, raise vad_threshold or interruption_delay, or enable Voice Focus.