How to build the lowest latency voice agent in Vapi: Achieving ~465ms end-to-end Latency

Every voice agent conversation is a race against a number most people never measure: how long the other person is willing to sit in silence before they think the line dropped.

That number is roughly 800 milliseconds. Past it, people start talking over your agent, repeating themselves, or hanging up. Under it, the conversation feels like a conversation.

Vapi gives you the orchestration to hit that. What it doesn't give you is a good default stack for a voice agent — you pick the transcriber, the LLM, and the voice, and those three choices decide whether your agent feels alive or feels like a bad conference call. This post is about making those choices well, and about the one setting that quietly ruins more Vapi agents than every other setting combined.

Fair warning: we're going to be stingy with numbers here. There's a lot of latency marketing in this space, and most of it compares a vendor's best-case partial-transcript timing against another vendor's worst-case final. We'll tell you exactly what we can stand behind and exactly what you have to measure yourself.

Where the milliseconds actually go

A Vapi agent is a relay race with four legs. Someone stops speaking. Your transcriber decides they're done and hands over a final transcript. Your LLM starts generating. Your TTS starts speaking. Audio reaches the caller's ear.

Every leg has a handoff cost, and the handoffs are where naive latency budgets fall apart. The number that matters isn't "how fast is your STT" — it's how long after the user's last syllable does the first audio of the reply arrive. Those are very different measurements, and only one of them is what your caller experiences.

Here's the honest budget:

Notice what's not in that table: a total. We're not going to publish a composite end-to-end figure for a stack we don't control end to end, because the moment you swap the voice or add a tool call, the number is fiction.

The one end-to-end number we will stand behind is our own: our Voice Agent API runs about 1 second from the end of the user's speech to the start of the agent's, over a single WebSocket. More on that at the end, because for a meaningful number of teams it's the right answer and this whole tuning exercise is a detour.

But if you're building on Vapi — and plenty of good reasons exist to — the rest of this is how you get the first two legs of that race as tight as they go.

Step 1: Configure the transcriber

This is the step the original version of this post skipped, which is a strange thing to skip in a build guide. So here's the actual configuration.

In the Vapi dashboard, go to Settings → Transcriber Providers, add your AssemblyAI API key, then create or edit an assistant and select Assembly AI as the transcriber with universal-3-5-pro as the model.

If you'd rather work in JSON — and for anything you intend to version-control, you should — the assistant's transcriber block looks like this:

{
  "transcriber": {
    "provider": "assembly-ai",
    "speechModel": "universal-3-5-pro",
    "mode": "min_latency",
    "languageCodes": ["en"],
    "keytermsPrompt": ["Vapi", "AssemblyAI", "Universal-3.5 Pro"],
    "agentContext": "Can I grab your account number, please?"
  }
}

Six fields. Here's what each one is doing for your latency budget.

speechModel: "universal-3-5-pro" selects Universal-3.5 Pro Realtime, our current streaming flagship and the speech foundation under our own Voice Agent API. It scores 6.99% pooled word error rate and 15.31% entity error rate on Pipecat's open STT benchmark, against 15.58%/50.50% for Deepgram Flux, 9.76%/39.70% for ElevenLabs Scribe v2, and 9.04%/21.51% for Google Chirp3. On phone numbers specifically it's at 3.55%. Those numbers come from the Universal-3.5 Pro Realtime launch post; see our benchmarks page for wider model comparisons.

If you're migrating an older Vapi assistant, this is the field to change first. universal-streaming-english is two generations back, and u3-rt-pro auto-routes to universal-3-5-pro as of August 7, 2026 — don't leave it in your config, because a pinned identifier that silently redirects is a config you'll misread six months from now.

mode is the whole latency tuning story now. We'll cover it in the next section.

languageCodes biases the model toward the languages you expect. Universal-3.5 Pro Realtime code-switches natively across 18 languages — including mid-sentence Hinglish — with no configuration at all, so you can omit this entirely for a multilingual agent. But if you know your callers speak English, passing ["en"] is free accuracy. If you know it's English and Spanish, pass both and keep the code-switching.

keytermsPrompt boosts recognition of names, brands, and domain terms. Up to 100 terms, 50 characters each, included at no extra cost. Put your product names, your competitors' product names, and any term a caller will say that a general-purpose model has never seen.

agentContext is the one people skip and shouldn't. You pass in what your agent just said, and the model uses it to resolve the reply. When your agent asks "what's your account number," the model knows a string of digits is coming and stops treating the pause between digit groups as the end of a turn. Across a benchmark of 20,000 voice agent audio files, passing agent context cut word error rate by 10.2% — with fabrications down 18.3% and short-utterance errors down 13.7%.

The model also keeps a rolling conversation memory — we call it Context Carryover — on by default, so it carries what was already said in the call without you managing it.

Step 2: Pick a mode instead of tuning flags

Older versions of this advice told you to reach into low-level flags — turn off formatting, drop confidence thresholds, shave silence timers. That whole approach is gone, and good riddance. It produced agents that were fast at being wrong.

Universal-3.5 Pro Realtime exposes three modes instead:

min_turn_silence is how long the model waits in silence before it runs an end-of-turn check. max_turn_silence is the hard fallback — the point where the turn ends regardless of whether the thought reads as finished. interruption_delay is how long after the user starts talking before a barge-in registers, which is your defense against "mm-hmm" cutting the agent off mid-sentence.

Here's the part worth internalizing: min_latency doesn't make transcription faster, it makes the model less patient. It's not buying you speed for free — it's trading a little tolerance for mid-thought pauses in exchange for a snappier turn. For a bot confirming a pizza order, that's a great trade. For a bot taking a callback number, it's a terrible one.

Which is why the best-performing Vapi agents don't pick a mode. They switch it.

Run min_latency for normal conversational back-and-forth. The moment your agent asks for something you can't afford to get wrong — an account number, an email address, a name being spelled out — switch to max_accuracy for the duration of that answer, then switch back. The mode is updatable mid-session, so it costs you a message, not a reconnect.

If you're driving our streaming WebSocket directly rather than through Vapi, the same thing looks like this:

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "mode": "min_latency",
}
API_ENDPOINT = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(CONNECTION_PARAMS)}"

# Entering an entity-capture step — the caller is about to read out a phone number
ws.send(json.dumps({"type": "UpdateConfiguration", "mode": "max_accuracy"}))

# Value captured — restore the preset
ws.send(json.dumps({"type": "UpdateConfiguration", "mode": "min_latency"}))

One thing you will not find here: an instruction to disable formatting. Formatting is always on in Universal-3.5 Pro and isn't a parameter anymore. It was never worth turning off anyway — handing your LLM my number is five five five oh one three four instead of My number is 555-0134. doesn't save you time, it just moves the parsing problem downstream where it costs more.

Step 3: Choose an LLM that streams fast, not one that scores well

Vapi lets you point at basically any model. The instinct is to pick the smartest one you can afford. For a voice agent, that's usually the wrong instinct.

What matters is time-to-first-token, because Vapi can start feeding TTS as soon as the first tokens arrive. A model that finishes generating in 400ms but doesn't emit its first token for 350ms will feel slower than a model that takes 900ms total but starts streaming at 120ms. Total generation time is nearly irrelevant. First-token time is everything.

Three things to get right regardless of which model you pick:

• Cap the output. Set a max token limit low enough that the model can't monologue. Voice replies should be one or two sentences; if the model wants to give you five, that's five sentences of TTS your caller has to sit through before they can respond.
• Write the system prompt for speech, not for text. "Answer in one or two sentences. Never use bullet points or lists." Costs you nothing, saves you seconds.
• Measure tool-call latency separately. A tool call is a full extra round trip that your latency budget probably doesn't account for. If your agent calls a slow internal API mid-conversation, no amount of STT tuning saves you.

We're deliberately not naming a model here. The last version of this post recommended a specific one with a specific millisecond figure, and both went stale inside a year. Benchmark two or three candidates against your own prompts and your own tools — how to evaluate speech recognition models covers the methodology, and most of it transfers directly to evaluating an LLM leg.

Step 4: Pick a voice that starts fast

Same logic, different leg. What you care about in TTS is time-to-first-byte — how quickly the first audio frame arrives — not how long the full utterance takes to synthesize, because Vapi streams it.

A few things reliably cost you TTFB and are worth checking before you blame your provider: heavy prosody or style-exaggeration settings, very long input strings that force the model to plan the whole utterance, and voices built for expressive narration rather than conversation. Conversational voices are usually faster and, for an agent, usually better anyway.

Our rundown of current text-to-speech APIs goes deeper on the tradeoffs. Whichever you choose, test it with your actual reply lengths — a voice that's fast on "Sure, one moment" can be considerably slower on a three-clause sentence.

Step 5: Let the transcriber own turn detection

This is the section that matters most, and it's the one where the old advice has aged worst.

Vapi has a startSpeakingPlan with a set of timers — waitSeconds, and under transcriptionEndpointingPlan, values like onPunctuationSeconds, onNoPunctuationSeconds, and onNumberSeconds. For years the standard advice, including in the original version of this post, was to go hand-tune those numbers down, because the defaults added well over a second of dead air before your agent would respond.

Don't do that anymore.

Vapi's smartEndpointingPlan now accepts assembly as a provider, and Vapi's own documentation is unambiguous about how to use it: choose AssemblyAI as your transcriber and don't set a separate smart endpointing plan. Vapi reads the end_of_turn flag we send and acts on it directly.

That's a meaningfully different architecture from timer-tuning. Universal-3.5 Pro Realtime doesn't endpoint on silence — it reads the transcript itself to judge whether someone finished a thought or just took a breath, at around 300ms. A human pausing to think mid-sentence leaves a different transcript from a human who's done talking, and the model can read the difference. A timer can't. That's why hand-tuned timers always land in the same trap: tight enough to feel responsive means you interrupt people, and loose enough to stop interrupting people means you feel slow.

Stacking your own timers on top of a model that's already making this decision doesn't add safety. It adds latency, and it overrides a better signal with a worse one. We went deep on why in how intelligent turn detection solves the awkward-pause problem.

The other half of the equation is barge-in. Our model emits a SpeechStarted event only once it produces actual transcript — background noise alone won't fire it — which makes it a reliable interrupt trigger. If your agent is cutting itself off, it's almost always picking up its own TTS through the mic; raise interruption_delay or vad_threshold before you touch anything else.

Accuracy is a latency feature

Here's a thing that doesn't show up in any latency budget and dominates real-world perceived speed: a misrecognition costs you an entire conversational turn.

If your agent hears "my number is 555-0134" as "my number is 555-01 34" and asks the caller to repeat it, you didn't lose 200 milliseconds. You lost eight seconds and some of the caller's patience. No amount of shaving the TTS leg gets that back.

This is why entity accuracy is the number to watch for voice agents, not pooled word error rate. Callers say phone numbers, email addresses, order IDs, and their own names — the exact categories general-purpose models handle worst. Universal-3.5 Pro Realtime sits at 15.31% entity error rate against 50.50% for Deepgram Flux and 39.70% for ElevenLabs Scribe v2, and that gap shows up in your agent as fewer "sorry, could you repeat that" loops.

Three settings that buy accuracy for free or near-free:

• agentContext — pass in what your agent just asked. Costs nothing, cut WER 10.2% across 20,000 voice agent files.
• keytermsPrompt — up to 100 domain terms, included in the base price.
• voice_focus — speaker isolation, near-field for headsets and phones, far-field for rooms, kiosks, and drive-thrus. +$0.10/hr, and worth it the moment your audio has a second human in it.

Base price for all of this is $0.45/hr on streaming speech-to-text, billed per second on session duration with unlimited concurrency and automatic rate-limit scaling. Diarization with revision is +$0.12/hr, prompting +$0.05/hr. Full breakdown on the pricing page.

Fireflies went through this evaluation for their own voice agent pipeline. Foysal Osmany, a software engineer there, put it plainly:

We were searching for the best realtime ASR model for our voice agent pipeline in Fireflies. The new Universal 3.5 Pro speech model from Assembly is best so far in terms of accuracy, latency and language switching.

Network and deployment

Two deployment realities that no config change fixes.

Region matters more than any single parameter. Every leg of the pipeline is a network hop, and if your Vapi region, your LLM region, and your transcriber region are on three different continents, you've spent more on geography than you'll ever recover in tuning. If you need EU data residency, we're at streaming.eu.assemblyai.com at the same price as US.

Telephony is a different animal from web. Phone audio arrives narrowband and adds carrier-side buffering you can't tune away. Test on the medium you'll ship on — a demo that feels perfect over a laptop mic can feel sluggish on a landline, and you want to find that out before your customer does.

Testing and monitoring

Component-level metrics or you're guessing. Log, per turn: time from the user's last syllable to end_of_turn, time from end_of_turn to first LLM token, and time from first token to first audio frame. If you only log total round-trip time, you'll optimize the wrong leg.

Then log the things that don't show up as latency at all but read as latency to the caller:

• Interruption rate — how often your agent cuts someone off. Rising interruptions means your mode is too aggressive.
• Repeat rate — how often callers repeat themselves. Rising repeats means either your agent is too slow or it's mishearing.
• Turn fragmentation — how often one utterance became two turns. This is the phone-number problem, and it's the tell that you need max_accuracy or a higher min_turn_silence at that step.

Test with real audio, not with your own voice in a quiet room. Your callers are in cars, in warehouses, and on speakerphone. Our 2026 insights report on what actually makes a good voice agent has more on where production agents break that lab testing never catches.

Common pitfalls

Still running universal-streaming-english. It's two generations old. If your Vapi assistant was configured before mid-2026, this is almost certainly what it's set to, and it's the single highest-value change in this post.

Hand-tuned startSpeakingPlan timers left over from an older build. They're now fighting the model instead of helping it. Clear them and let the transcriber's end_of_turn drive.

Chasing STT milliseconds while a tool call eats a second and a half. Instrument every leg before you optimize any leg.

min_latency everywhere. It's the right default for rapid back-and-forth and the wrong default for anything involving a number, an address, or a name. Switch modes by call stage.

Long system prompts. They inflate time-to-first-token on every single turn. Cut them.

Testing only on clean audio. See above. Your agent's worst latency day is the day someone calls from a car.

When to stop assembling a stack

Everything above assumes you want the control that comes with picking your own transcriber, LLM, and voice. That's a real preference and Vapi serves it well.

But it's worth naming the alternative honestly, because some of the teams reading this are optimizing a stack they'd rather not be maintaining. Our Voice Agent API is one WebSocket at wss://agents.assemblyai.com/v1/ws that does STT, LLM, and TTS in a single connection — flat $4.50/hr, ~1 second end-to-end, built on the same Universal-3.5 Pro Realtime you'd be configuring in Vapi. Turn detection and barge-in are on by default with nothing to tune. No SDK required, and it works with Claude Code out of the box.

We're infrastructure, not an agent platform, so this isn't a Vapi replacement for everyone — if you need Vapi's orchestration, campaign tooling, or provider flexibility, keep it and configure the transcriber well. But if the reason you're reading a latency-tuning guide is that you've been hand-maintaining a four-vendor pipeline, the honest advice is to price out the single-connection version before you spend another sprint on timers. The production ceiling covers where assembled stacks tend to hit their limits.

Final words

The hardest thing to accept about voice agent latency is that the fastest agent isn't the one that wins.

The agent that wins is the one that's predictably fast — the one that responds in a consistent rhythm the caller can subconsciously sync to. A stack that averages 700ms but swings between 300ms and 1,800ms feels worse than one that lands on 900ms every single time, because humans calibrate to a rhythm and get thrown by variance far more than by absolute delay. It's the same reason a metronome at 60bpm is easier to follow than one that averages 60 but wanders.

So when you instrument your agent, log the p90 and p99, not just the mean. Your mean is what you put on a slide. Your p99 is what your caller remembers.

And most of the variance won't be in the legs you were tuning. It'll be in the turn detection — in how consistently the transcriber decides someone is done talking. That's the one that's least visible in a latency dashboard and most audible on a phone call.

Frequently asked questions

What is the lowest latency speech-to-text model for a Vapi voice agent?

Universal-3.5 Pro Realtime (universal-3-5-pro) in min_latency mode is the fastest AssemblyAI configuration available in Vapi. In that mode the model runs an end-of-turn check after 128ms of silence and force-ends a turn at 640ms, with end-of-turn detection landing around 300ms and a complete formatted transcript under 300ms after that. Configure it in the Vapi dashboard under Settings → Transcriber Providers, or set "speechModel": "universal-3-5-pro" in the assistant's transcriber block.

How do I configure AssemblyAI as the transcriber in Vapi?

Add your AssemblyAI API key in the Vapi dashboard under Settings → Transcriber Providers, then set your assistant's transcriber to "provider": "assembly-ai" with "speechModel": "universal-3-5-pro". From there, mode, languageCodes, keytermsPrompt, and agentContext are the four fields worth setting. There's a full walkthrough in our Vapi voice agent guide.

Should I tune Vapi's startSpeakingPlan settings for lower latency?

No — not when AssemblyAI is your transcriber. Vapi's documentation says to choose AssemblyAI as the transcriber without setting a separate smart endpointing plan, because Vapi reads the end_of_turn flag the model sends. Universal-3.5 Pro Realtime judges the end of a turn from the transcript itself rather than a silence timer, so hand-tuned wait times override a better signal with a worse one and add latency doing it.

What's the difference between min_latency, balanced, and max_accuracy mode?

They're presets that set the model's silence thresholds and interruption delay: min_latency is least patient (128ms before an end-of-turn check, 640ms hard cutoff, 0ms interruption delay), balanced is the default middle ground (128ms / 1280ms / 500ms), and max_accuracy waits longest (512ms / 2560ms / 500ms). They replace the low-level flags older streaming models exposed. Mode is updatable mid-session, so the strongest pattern is running min_latency for normal conversation and switching to max_accuracy while capturing a phone number or an email address.

How much does real-time transcription cost for a voice agent?

Universal-3.5 Pro Realtime is $0.45/hr base, billed per second on WebSocket session duration, with unlimited concurrency and automatic rate-limit scaling. Agent context, rolling conversation memory, and keyterm prompting are included; diarization with revision is +$0.12/hr, prompting +$0.05/hr, and voice isolation +$0.10/hr. If you'd rather not assemble STT, LLM, and TTS separately, the Voice Agent API bundles all three through one WebSocket at a flat $4.50/hr.

Can a Vapi voice agent handle more than one language?

Yes. Universal-3.5 Pro Realtime code-switches natively across 18 languages — including mid-sentence, and including Hinglish — with no configuration and no separate multilingual model. Omit languageCodes entirely for full multilingual behavior, or pass the codes you expect (like ["en", "es"]) to bias the model toward them while keeping code-switching intact. Supported languages are English, Spanish, French, German, Italian, Portuguese, Arabic, Danish, Dutch, Hebrew, Hindi, Japanese, Mandarin, Vietnamese, Finnish, Norwegian, Swedish, and Turkish.