Now live New models for transcription and text-to-speech. AssemblyAI-STT-3.7 AssemblyAI-TTS-1
Insights & Use Cases

The 300ms rule: Why latency makes or breaks voice AI applications

Low latency voice AI keeps the speech-to-text layer inside a 300 ms conversational pause. Where the milliseconds go, and which AssemblyAI API to use for each job.

Abstract green cylinder illustration

Written by

Martin Schweiger

Published on

19 November 2026

The rule of thumb in voice AI is that your speech-to-text layer should return a complete transcript in about the length of a natural conversational pause: roughly 300 milliseconds. That number comes from how people take turns in conversation. Go past it and the person on the other end starts to feel the machine. But 300 ms is a budget for one layer, and treating it as the budget for a whole interaction is the most common mistake in vendor comparisons. This post separates the two: what each layer costs you in milliseconds, which AssemblyAI low-latency API to reach for, and the published numbers for each.

The short version: the Sync API returns a finished short-clip transcript in one HTTP response (~134 ms p50 at its launch). The Dictation API returns cleaned-up text, with typical short clips back in under one second. AssemblyAI-STT-3.7-Realtime-Agent streams transcripts as audio arrives and returns the final transcript 86 ms after the caller stops speaking. A full agent loop built from it, an LLM and AssemblyAI's Text-to-Speech API has to fit a complete conversational turn inside about one second.

Why 300 milliseconds is the number

Human conversation runs on a tighter clock than most people realize. The gap between one speaker finishing and the next starting sits, across languages and cultures, in the low hundreds of milliseconds. Listeners start planning a reply before the other person's sentence ends, predicting the ending while it's still being spoken. That's why a natural conversation can have turn gaps shorter than the time it takes to say a single word.

The practical consequence is that people don't experience latency in voice AI linearly. Under about 300 ms, a delay reads as ordinary conversational rhythm and nobody notices it. Somewhere past 500 ms it reads as hesitation, and the listener starts to wonder whether they were heard. Past a second, people repeat themselves, which produces overlapping speech, which produces exactly the kind of audio that degrades transcription accuracy and makes the problem worse. Latency failures compound.

So the design goal is to stay inside the window where the delay is invisible, and spend the rest of your budget on accuracy.

Where the milliseconds go

A voice interaction is a chain, and every link spends time:

  • Capture and network transit. Getting audio off the device and to the API. Typically tens of milliseconds, more on a poor mobile connection, and largely out of your control. Picking a region close to your users helps more than most optimizations you can make in code.
  • Endpointing. Deciding the speaker has finished. This is often the largest and least examined cost in the chain. A naive implementation waits out a fixed silence window, 700 ms or a full second, before it commits to a turn ending, and that wait lands entirely in the user's experience.
  • Transcription. Turning audio into a finalized transcript. This is the layer the 300 ms rule is about.
  • Reasoning. Whatever your application does with the text: a model call, a database lookup, a tool invocation.
  • Synthesis. Turning the response back into audio, and the time to the first audible byte.

Most teams optimize transcription because it's the layer with a published number, then wonder why their agent still feels slow. The usual culprit is endpointing. AssemblyAI's streaming models don't endpoint on silence alone. When a speaker pauses, the model evaluates what has been said so far and judges whether it reads as a finished thought or a mid-thought pause. If the turn ends in something that may continue, such as part of a phone number, the turn stays open. A speaker who trails off mid-sentence and a speaker who has finished produce text that reads very differently even when the silence is identical, and a fixed timer can't tell them apart.

Hear The Difference Before You Architect

Run your own audio through the current models and listen to where the milliseconds go. No setup, no integration work.

Try playground

The low-latency APIs, compared

AssemblyAI publishes three low-latency paths, plus the full agent loop you compose from them. They're different shapes, and picking the wrong one is the most common source of "why is this slow" tickets.

API Published latency Shape Price Reach for it when
Sync API ~134 ms p50 on a 2-second clip at launch, request to finished transcript (versus 5–6 s on async submit-and-poll) Single HTTP request, single response $0.30/hr You have a short clip and you want the transcript itself, right now
Dictation API Typical short clips respond in under one second, transcription and cleanup included Single HTTP request; transcription plus an LLM cleanup pass $0.62/hr The text goes straight into whatever your user is writing
AssemblyAI-STT-3.7-Realtime-Agent Partials as audio arrives; final transcript 86 ms after the caller stops speaking Persistent WebSocket, continuous audio $0.45/hr base Audio is open-ended and something has to answer it
Full agent loop (Realtime-Agent + LLM + Text-to-Speech API) About one second end to end is the budget; AssemblyAI's own stack measured around 250 ms per turn Streaming WebSocket in, LLM call, text-to-speech out; LiveKit or your own loop carries the turn $0.45/hr base for speech-to-text, plus LLM and text-to-speech usage You're building something that talks back

If your live audio only needs to be displayed, as captions or meeting notes, AssemblyAI-STT-3.7-Realtime ($0.30/hr) is the streaming model built for that. Its turn detection segments the transcript instead of driving turn-taking, so for anything that replies, use the Agent model.

The Sync API is the path most short-clip features should have started with

If your audio is short and bounded (a push-to-talk clip, a voice command, an IVR response, a voicemail), you don't need a WebSocket. You need one request and one response. That's the Sync API: POST https://sync.assemblyai.com/v1/transcribe with the audio, and the finished transcript comes back in the same response. At launch it measured ~134 ms p50 from request to finished transcript on a 2-second clip.

The comparison that matters is against the async submit-and-poll pattern a lot of dictation features were built on by default: submit a job, poll for completion, render the result. That path takes 5 to 6 seconds for the same clip. For a dictation box or a push-to-talk button, that's the difference between a feature people use and a feature people work around.

Sync runs Universal-3.6 Pro, selected with the X-AAI-Model: universal-3-6-pro header, and covers 32 languages. Over HTTP, /v1/transcribe takes audio from 80 ms to 20 minutes and up to 256 MB, in WAV, raw PCM or a compressed format such as MP3 or Ogg/Opus. The SDKs send through /v1/transcribe/live, which takes up to 120 seconds of WAV or raw PCM and can start while the audio is still being recorded. For the lowest latency, send an unauthenticated GET /v1/warm as soon as you know audio is coming, so the transcription request reuses an open connection. Sync returns a transcript; for speaker labels, PII redaction or Speech Understanding, use the pre-recorded or streaming APIs. The getting-started guide has a working request, and the Sync launch post has the launch numbers.

The Dictation API saves you a hop you'd otherwise pay for

The Dictation API is transcription plus an LLM cleanup pass in a single low-latency call. Typical short clips respond in under one second, rewrite included.

Transcription models are verbatim by design, and what someone said isn't what they want to send. Given "um so can we uh move the the meeting to thursday i think friday works better actually", the Dictation API returns "Can we move the meeting to Friday? That works better." Self-corrections resolve to what the speaker landed on, filler is gone, and the tone stays intact.

The alternative is doing that cleanup yourself, which means a second network hop to an LLM after the transcript comes back. That hop costs more than the headroom you're trying to protect, plus a second line on your bill and a second failure mode to handle. Dictation is $0.62/hr with transcription and cleanup included. It runs on Universal-3.6 Pro across 32 languages, and rewrite_mode picks between fast (the default, lowest latency) and smart (a more thorough cleanup with higher latency).

The clean way to choose between the two: reach for Sync when you want the transcript itself, and Dictation when the text is going straight into whatever your user is writing. The Dictation API launch post covers the rest, and the dictation documentation has the configuration fields.

AssemblyAI-STT-3.7-Realtime-Agent is the streaming path, and it's where the 300 ms rule lives

When audio is open-ended (a call in progress, a support line, a kiosk), you want a persistent connection and transcripts as the audio arrives. For anything that talks back, that's AssemblyAI-STT-3.7-Realtime-Agent, over wss://streaming.assemblyai.com/v3/ws with speech_model set to assemblyai-stt-3-7-realtime-agent, at $0.45/hr base.

On latency, the AssemblyAI-STT-3.7-Realtime-Agent research post reports the final transcript 86 ms after the caller stops speaking, half the 172 ms of Universal-3.6 Pro Realtime measured on the same day. That's well inside the 300 ms budget, which leaves room for the reasoning and synthesis layers.

The controls that matter for latency work on this API:

  • Mode. min_latency, balanced (the default) and max_accuracy set the turn-detection defaults, so change the mode before you touch anything else. Most voice agents should stay on balanced. The Agent model also has a manual mode, where your client ends each turn with ForceEndpoint, for teams running their own VAD, push-to-talk or turn-detection model.
  • Content-aware end-of-turn detection. When a speaker pauses, the model evaluates what has been said so far instead of counting silence alone. Entity-aware endpointing is on by default in every mode: a turn that ends in a phone number, an email address or a code is held open until max_turn_silence, so your LLM doesn't receive the number in pieces.
  • Turn detection parameters. min_turn_silence and max_turn_silence follow the mode preset: 128 / 1280 ms on balanced, 128 / 640 on min_latency, 512 / 2560 on max_accuracy. vad_threshold defaults to 0.2, and interruption_delay is 0 in every mode on the Agent model. Enabling speaker_labels swaps in diarization-tuned values of 640 / 768, so re-measure after you turn it on. When you know an entity is coming, raise min_turn_silence mid-stream with UpdateConfiguration, then send your mode again to restore the preset. Hard-coded numbers on the way out leave the session off-preset.
  • agent_context passes your agent's most recent spoken reply, so short and mumbled answers resolve against what was asked. Context carryover and conversation memory are built in.
  • Voice Focus conditions the audio before the model: near-field for headsets and phones, far-field for rooms, kiosks and drive-thrus, with voice_focus_threshold from 0.0 to 1.0 (default 0.5). It's a +$0.10/hr add-on.
  • language_codes as a list steers toward specific languages. Omit it for automatic detection and native code-switching across 32 languages.
  • Diarization with speaker_labels: true, capped with max_speakers (1 to 10). If you need speaker labels in a live transcript, streaming speaker diarization covers the tradeoffs.

Note the parameter name: streaming takes speech_model, singular. The pre-recorded API takes speech_models, plural, as a list. It's a small thing that produces a confusing error.

Speed only helps if the transcript is right. On AssemblyAI's run of 12,460 English voice-agent calls across three noise conditions, AssemblyAI-STT-3.7-Realtime-Agent has the lowest word error rate of the 26 streaming models we ran, 4.53%, ahead of Soniox Realtime v6 (6.02%), ElevenLabs Scribe v3 Live (7.38%) and Deepgram Nova-3 (8.64%). Its entity error rate is 13.2%, against 15.2% for Soniox Realtime v6, 16.9% for ElevenLabs Scribe v3 Live and 25.1% for Deepgram Nova-3. Full tables are on the benchmarks page.

If you're still on Universal-Streaming or Universal-3.5 Pro Realtime, both retire on February 1, 2027. Migrate new and existing voice-agent builds to AssemblyAI-STT-3.7-Realtime-Agent; the launch announcement covers the rest of the family.

Where the full agent loop fits, and why its number is bigger

The 300 ms rule and the one-second figure for a full agent turn measure different spans. Vendor comparisons in this category routinely put the two next to each other as if they were the same measurement, so it's worth being precise.

The 300 ms budget is a speech-to-text measurement: from the moment the speaker stops to the moment a complete, finalized transcript exists. It's one layer.

About one second is an end-to-end conversational measurement: from the moment the caller stops speaking to the moment the agent starts speaking. That span contains the transcription, plus the reasoning step, plus speech synthesis, plus the network transit for all of it. It's the outer limit before a turn starts to feel slow, and a well-built loop lands well under it. In the research post, AssemblyAI-STT-3.7-Realtime-Agent with Gemma 4 31B through LLM Gateway and AssemblyAI-TTS-1 measured end-to-end turn latency regularly around 250 ms.

A vendor quoting 200 ms and a vendor quoting one second may be describing systems with identical felt latency, or wildly different ones. The only way to know is to ask which span they're measuring. When you benchmark, measure the end-to-end span yourself, because it's the only one your users experience.

Three models run inside a full loop: AssemblyAI-STT-3.7-Realtime-Agent for recognition, an LLM through LLM Gateway for reasoning, and AssemblyAI's Text-to-Speech API for the response, with LiveKit or your own loop carrying the turn. (Pipecat's AssemblyAI service, pipecat-ai 1.16.0 on PyPI, doesn't accept the 3.7 models yet, so Pipecat pipelines run Universal-3.6 Pro Realtime for now.)

Consolidating vendors is a latency argument. When you assemble an agent from three vendors, you pay network transit between each pair, you pay serialization at each boundary, and you own the reconnection logic for three connections that can each fail independently. Taking both speech ends from one vendor doesn't remove the hops, but it puts them on one status page and one set of logs when the tail moves. Those costs don't show up in anyone's published per-layer number. They show up in your p99.

Two things move end-to-end latency once the speech layer is tuned. First, stream every stage: hand the finalized turn to the LLM the moment it lands, and let text-to-speech start synthesizing the first sentence while the LLM is still writing the second. LiveKit and Pipecat both do this by default. Second, leave the turn-detection parameters on their mode preset unless you've measured a reason to move them. Hard-coded silence windows are how agents cut a caller off in the middle of a phone number. agent_context plus clear tool descriptions improve turn-taking more than VAD knobs do.

Pricing follows a simple shape too: AssemblyAI-STT-3.7-Realtime-Agent is $0.45/hr base, billed on WebSocket session duration, and text-to-speech lands on the same invoice at $15 per 1M characters. If you're coming from a stack where you model three vendors' bills to answer "what does a call cost," the speech layer is now one line.

The seconds at the end of every call that you're probably paying for

One detail almost nobody catches on the first build: streaming bills on session duration, open to close, not on speech. A WebSocket left open after the caller hangs up, while your orchestrator tears down or a reconnect timer runs, is billable silence.

Send the Terminate message the moment a call is intentionally over, and let your orchestrator's reconnect logic handle unintentional drops. Terminate on intentional hangups; reconnect on everything else.

Start Building On The Low-Latency Stack

New accounts get $50 in free credits for the speech APIs, no credit card required (LLM Gateway isn't included). That's enough to measure end-to-end latency on your own traffic before you commit to an architecture.

Sign up free

Where the LLM layer costs you time, and how to get it back

If you do anything with the transcript beyond displaying it (cleanup, summarization, formatting, routing), the model call is part of your latency budget, and it's usually the largest single item in it.

LLM Gateway gives you 55+ models behind one endpoint, at llm-gateway.assemblyai.com/v1/chat/completions in the US and llm-gateway.eu.assemblyai.com/v1/chat/completions in the EU. For latency work, the relevant pieces are automatic fallback to up to two backup models with a 500 ms retry default, per-fallback overrides for prompt, temperature and max tokens, prompt caching for Anthropic and OpenAI models, and multi-region failover across the US and EU. A slow provider stops being your outage. LLM Gateway usage isn't covered by the free speech credits.

For fast rewrite work (dictation cleanup, transcript rewriting, live formatting, turn summarization), Qwen3.5 4B Fast is self-hosted by AssemblyAI on our own GPUs with a latency-optimized 32k context window. The Qwen3.5 4B post reports it averaging 612 ms on voice rewrite tasks, at $0.10 per million prompt tokens and $0.50 per million completion tokens. Reintroducing LLM Gateway covers the rest of the platform.

A related piece of architectural advice saves more latency than any model choice: don't make a voice agent call a tool to extract, summarize or classify what was just said. That adds a model round trip inside the conversational loop. Collect the transcript events and run a single LLM Gateway call against the finished transcript when you need the structured output. The voice loop stays fast, and you can use a bigger model for the extraction step.

If your application only needs cleanup on short dictated audio, skip the second hop entirely and use the Dictation API, which does the transcription and the rewrite in one call.

A practical checklist for cutting latency

  1. Measure end to end, not per layer. Instrument from "user stopped speaking" to "system started responding." Per-layer numbers hide the transit and serialization costs between layers, and that's often where the time is.
  2. Look at endpointing before you look at the model. If you're waiting out a fixed silence window, you're probably spending more there than in transcription. On AssemblyAI's streaming models, the fastest fix is usually to stop hard-coding the silence windows and let the mode preset do its job. See turn detection in the docs.
  3. Match the API shape to the audio shape. Short bounded clips don't need a WebSocket. Open-ended audio does. Using streaming for push-to-talk adds connection setup to every interaction for nothing.
  4. Eliminate hops before you shave milliseconds. Removing one network round trip usually beats every micro-optimization you can make inside a layer.
  5. Deploy near your users. EU traffic should hit api.eu.assemblyai.com and streaming.eu.assemblyai.com: same price, and the data stays in the EU.
  6. Don't stack noise suppression. If you capture in a browser, leave noiseSuppression off and let server-side Voice Focus do the work. Two independent passes eat real speech and cost you accuracy, which costs you clarification turns, the slowest thing in any conversation.
  7. Flush your playback buffer on barge-in. Otherwise the user hears another second of stale speech after they interrupt, which reads as lag even though no API was slow.
  8. Optimize p99, not p50. Your median user has a fine experience on almost any architecture. The tail is where people hang up.
Working Against Hard Requirements

Self-hosted deployment, EU data residency, or a Business Associate Addendum (BAA) for PHI: talk it through with someone who has shipped low-latency voice systems under those constraints.

Talk to AI expert

Frequently Asked Questions (FAQs)

What is a good latency target for voice AI?

Roughly 300 milliseconds for the speech-to-text layer, and no more than about one second end to end for a full conversational turn. The 300 ms figure approximates the natural gap between turns in human conversation, which is why delays under it go unnoticed. Past roughly 500 ms people begin to hear hesitation, and past a second they start repeating themselves.

How fast is AssemblyAI's streaming speech-to-text for voice agents?

AssemblyAI-STT-3.7-Realtime-Agent returns the final transcript 86 ms after the caller stops speaking, per AssemblyAI's research post, half the 172 ms of Universal-3.6 Pro Realtime measured on the same day. With Gemma 4 31B through LLM Gateway and AssemblyAI-TTS-1, the same post reports end-to-end turn latency regularly around 250 ms.

What is the difference between the Sync API and the Dictation API?

The Sync API returns a verbatim transcript of a short clip in a single HTTP response. The Dictation API takes the same kind of utterance and returns what the speaker meant to send, with self-corrections resolved and filler gone, with typical short clips back in under one second. Both run on Universal-3.6 Pro across 32 languages. Reach for Sync when you want the transcript itself, and Dictation when the text goes straight into whatever your user is writing.

How fast is the Sync API compared to submitting an async job?

At launch, the Sync API measured about 134 ms at p50 for a 2-second clip, versus 5 to 6 seconds for the same clip through async submit-and-poll. Over HTTP it handles audio from 80 ms to 20 minutes and up to 256 MB; the SDKs take up to 120 seconds of WAV or raw PCM. It costs $0.30/hr.

Which streaming model should I use for a new voice agent?

AssemblyAI-STT-3.7-Realtime-Agent (assemblyai-stt-3-7-realtime-agent), at $0.45/hr base over wss://streaming.assemblyai.com/v3/ws. It covers 32 languages with native code-switching, decides end of turn from what has been said with entity-aware endpointing on by default, and takes agent_context and Voice Focus. Universal-Streaming and Universal-3.5 Pro Realtime retire on February 1, 2027.

Does adding an LLM step break the latency budget?

It doesn't have to, but once it's there it's usually the largest single item in the budget. Stream the finalized turn to the model, start text-to-speech on the first sentence, and keep extraction and summarization out of the conversational loop. For short dictated audio, the Dictation API does transcription and cleanup in one call and removes the hop entirely.