The voice AI stack for building agents in 2026

Most voice agent projects don't fail on the model. They fail in the seams between models. You pick a good speech-to-text API, a good LLM, a good TTS voice, wire them together, and the demo is charming. Then you put it on a real phone line with a real caller who has an accent, a barking dog, and an account number to read out, and the whole thing falls apart — not because any single component is bad, but because each one adds a few hundred milliseconds and a few percent of error, and those compound.

McKinsey's State of AI found 40% of respondents at large organizations now scaling AI agents, up from 27% a year earlier. A lot of those teams are about to learn the same lesson: the stack is the product. So here's what the stack actually looks like in 2026, layer by layer, with the numbers that matter and the tradeoffs nobody puts in the marketing copy.

Deconstructing the modern voice agent stack

Every voice agent, no matter how it's packaged, does four things in a loop. It converts speech to text. It decides what to say. It converts text back to speech. And it decides when to talk — which is the part everyone underestimates.

Those four jobs map to speech-to-text, an LLM, text-to-speech, and orchestration. What varies between architectures is how many vendors and network hops sit between them, and how much of the turn-taking logic you own versus inherit.

The budget is fixed regardless. Human conversational turn-taking runs on a gap of a couple hundred milliseconds. Push past about 1.5 seconds of silence before your agent responds and callers start talking over it, apologizing, or hanging up. Every layer you add spends from the same account.

Voice AI agent architecture

Think of the pipeline as a latency budget with four line items, and be honest about each one.

Speech-to-text needs to decide the caller has finished — that's endpointing, and it's usually the largest single cost. The LLM needs to produce its first token. Text-to-speech needs to produce its first audio byte. And the network needs to move all of it, twice.

Here's the trap: teams optimize the three model calls and ignore endpointing. But a system that waits a fixed 800 ms of silence before deciding the turn is over has already spent more than half the budget before a single model runs. Turn detection is where the latency actually lives, which is why we've written a full breakdown of voice agent turn detection separately.

Voice AI architecture patterns

There are four shapes worth knowing. Three of them were in this post last year; the fourth is new and quietly useful.

Cascading pipelines

Record audio, send the whole utterance to speech-to-text, get a transcript, send it to an LLM, get text, send it to TTS, play the result. Each stage waits for the previous one to finish.

It's the easiest thing to build and the easiest thing to reason about, and it's why almost every tutorial starts here. It's also the slowest. Sequential handoffs put you at 1.6 seconds before the caller hears anything, and 1.5 seconds is roughly where a conversation stops feeling like a conversation. Fine for a voicemail transcriber. Rough for a phone agent.

Streaming architectures

Same components, but nothing waits. Audio streams into speech-to-text continuously, partial transcripts flow to the LLM as they stabilize, and TTS starts speaking the first sentence while the LLM is still writing the second.

Done well, this gets you under 1,500 ms total from end-of-speech to first audio, with roughly 500 ms per stage. It's more work — you're managing partials, handling revisions, and deciding when a partial is stable enough to act on — but it's the pattern production agents converge on. Universal-3.5 Pro Realtime is built for it.

Single-request synchronous calls

This is the lane that's missing from most stack diagrams, and it fits a real class of agent.

If your application already knows when the user is done talking — a push-to-talk button, a dictation field, an IVR that plays a beep, or an agent running its own turn detection off a voice activity detector — you don't need a streaming session at all. You need one HTTP POST with a short clip and a finished transcript in the response.

That's the Sync Speech-to-Text API: POST https://sync.assemblyai.com/transcribe with an X-AAI-Model: universal-3-5-pro header, roughly 134 ms at p50 on a two-second clip, against five to six seconds for a submit-and-poll round trip. Clips run 80 ms to two minutes, up to 40 MB, WAV or raw PCM at 16 kHz by default, across 19 languages.

‍

curl -X POST https://sync.assemblyai.com/transcribe \
  -H "Authorization: <YOUR_API_KEY>" \
  -H "X-AAI-Model: universal-3-5-pro" \
  -F "audio=@utterance.wav;type=audio/wav"

The audio goes up as a multipart audio part, so let curl set the boundary — don't force a request-level Content-Type. If you're sending raw PCM rather than WAV, add a second part describing it: -F 'config={"sample_rate":16000,"channels":1};type=application/json'. The response comes back with text, words, confidence, audio_duration_ms, and request_time_ms.

The architectural appeal is that there's no session to manage, no WebSocket to reconnect, and no partial-transcript state machine. Your agent owns turn-taking; the transcription is a function call. Keyterms and conversation_context are included.

What you give up: Sync doesn't support PII redaction, speaker diarization, Speech Understanding, or Medical Mode. If your agent needs any of those inline, you're on the streaming path. The Sync API launch post covers the full surface.

End-to-end models

One model in, audio out, no intermediate text. Conceptually elegant. In practice, current end-to-end speech models land at 800–2000 ms of latency, which rules them out for live conversation, and you lose the transcript — which means you lose the thing your compliance team, your analytics, and your QA process all depend on.

Worth watching. Not worth betting a contact center on this year.

Speech-to-text: the foundation

Everything downstream inherits this layer's errors. An LLM given a mangled transcript will confidently reason about the wrong thing, and TTS will say it in a lovely voice.

Which is why the benchmark that matters is one built on real agent conversations, not clean read speech. Here's Universal-3.5 Pro Realtime on Pipecat's open speech-to-text benchmark, which does exactly that.

Read the second row before the first. Google Chirp3 is within about two points of us on pooled word error rate and roughly 40% worse on entities — which is the difference between an agent that hears "four-one-five" and one that hears "four-one-nine." Word error rate treats every word as equally important, and your callers do not. Entity accuracy is the number to shop on.

Three capabilities in Universal-3.5 Pro Realtime exist specifically because of how voice agents fail:

• agent_context — pass the agent's own question so the model hears the reply through its lens. If your agent just asked "what's your date of birth," the model expects a date. Across a benchmark of 20,000 voice agent audio files, passing agent context cut word error rate by 10.2%.
• Context Carryover — rolling conversation memory, on by default, nothing to configure. The model remembers that the caller said "Kowalczyk" three turns ago and spells it the same way this turn.
• Voice Focus (voice_focus, +$0.10/hr) — near-field for headsets, far-field for speakerphones and car audio, with a voice_focus_threshold defaulting to 0.7.

The rest of the sheet: 18 languages with mid-sentence code-switching, Hinglish included, worth a 22% relative reduction in word error rate on code-switched speech and another 4% with prompts. Live speaker diarization at +$0.12/hr sends a single re-clustering correction within about half a second of the stream ending, for up to ten speakers. Keyterms are included on the streaming flagship, capped at 100 terms of 50 characters each, and updatable mid-stream with UpdateConfiguration — so an agent can inject the caller's account details into the vocabulary the moment it identifies them.

Pricing is $0.45/hr base, billed on session duration, with add-ons that stack only if you use them. If you need coverage beyond those 18 languages, Universal-2 handles 99+ at $0.15/hr — the right call for long-tail language support and legacy integrations, not for a latency-sensitive agent.

One config note that has bitten more teams than it should. Set speech_model explicitly. A config that omits it rides whatever the account default happens to be, which is exactly the kind of thing that quietly changes underneath you.

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "mode": "balanced",
    "voice_focus": "near-field",
    "agent_context": "Thanks for calling — what can I help you with today?",
}

mode takes min_latency, balanced, or max_accuracy. Start at balanced; move only when you've measured a reason to. The full parameter surface is in the streaming model selection docs.

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms."

— David Zhao, Co-founder at LiveKit

Text-to-speech: giving voice to AI agents

TTS is the layer users judge you on and engineers think about last.

Four things matter. Time to first audio byte, where under 200 ms keeps the conversation feeling live. Mean opinion score, where above 4.0 is genuinely human-like. Prosody and emotional range, which is what separates "clearly a robot" from "possibly a person having a bad day." And streaming chunk support, because a voice that won't start speaking until the full sentence is generated has silently added a second to every turn.

Cartesia, ElevenLabs, and Rime all ship production-grade streaming voices. Pick on latency and prosody for your specific content — a voice that sounds great reading marketing copy can fall apart reading a 16-digit order number, and the only way to find out is to make it read your order numbers.

LLMs: the brain of voice AI agents

The instinct is to reach for the biggest model. Resist it.

In a voice agent, four things matter more than raw reasoning: time to first token, brevity under instruction, reliable function calling, and holding context across turns without drifting. A model that produces a brilliant six-sentence answer has failed, because nobody wants to listen to six sentences from a phone agent.

Gemini 3 Pro and GPT-5.2 are the right choice when the agent has to reason — multi-step troubleshooting, policy interpretation, anything where being wrong is worse than being slow. For the latency-critical path, smaller models like Gemini 2.5 Flash-Lite or Claude 4.5 Haiku are usually the better trade, and most production agents end up tiering: a fast model for the common path, an escalation to a larger one when the conversation gets hard.

That tiering is easier when you're not maintaining four vendor integrations. LLM Gateway gives you one API across Anthropic, OpenAI, Google, and Qwen — 33 models currently — with automatic cross-provider fallback, streaming with tool calling, structured JSON, and prompt caching. Model swaps become a string change instead of a sprint.

Orchestration

Something has to own the loop: audio in, turn detection, transcript to LLM, tokens to TTS, audio out, interruption handling, and the telephony underneath all of it. You have two real options.

Build on a framework

Vapi, LiveKit Agents, and Daily's Pipecat each give you the plumbing for building AI voice agents while leaving component choice to you. You get maximum control and you own the integration surface — every component upgrade, every vendor incident, every edge case in turn-taking is yours.

AssemblyAI ships drop-in plugins for LiveKit and Pipecat, so swapping the speech-to-text layer is a config change rather than a refactor. That matters more than it sounds: the ability to A/B two speech models on production traffic without a code change is how you actually find out which one is better on your audio.

Use a managed voice agent API

AssemblyAI's Voice Agent API is one WebSocket — wss://agents.assemblyai.com/v1/ws — that bundles speech-to-text, an LLM, and TTS behind a single connection at a flat $4.50/hr. It runs on Universal-3.5 Pro Realtime for speech, lands around 1 second end-to-end, and works with Claude Code out of the box with no SDK required.

Underneath it's still a cascade — speech-to-text into a Voice Agent LLM tuned for spoken conversation rather than text chat, into Voice Agent TTS — but the hops happen inside one persistent socket instead of across three vendors' networks. It is not a unified speech-to-speech model, and we'd rather say that plainly than let the architecture be misread.

Every feature is included in the $4.50/hr rate. There are no per-layer add-ons, concurrency fees, or per-agent subscriptions. Prompting, Voice Focus, advanced turn detection, interruption detection that ignores backchannels like "mhm" and "right," recordings and transcripts, infrastructure hosting, and bring-your-own-Twilio SIP trunking with no per-minute markup are all in the base rate. Audio is PCM16 at 24 kHz, and sessions are preserved for 30 seconds after a disconnect, so a dropped mobile connection can resume with session.resume and the prior session_id instead of starting the conversation over. That 30-second window is billable, so send session.end when a call is genuinely finished.

The Voice Agent API is infrastructure. It doesn't have opinions about your conversation design; it gives you the pipeline as a primitive and gets out of the way. That's not the same product category as no-code agent builders like Bland or Synthflow, which sell you a finished agent and a dashboard. Different thing, different buyer, and a reasonable choice if you want a working phone agent by Friday and never intend to touch the internals.

What the first week actually feels like

Developer experience is the least measurable thing in this post and the most decisive. Teams don't usually switch stacks because of a benchmark. They switch because something took three days that should have taken twenty minutes.

So here's what the first week looks like, concretely.

You can get audio flowing before you talk to anyone. Sign up, get 185 hours of pre-recorded transcription and 333 hours of streaming free, and start. No sales call, no minimum commitment, no contract, no procurement gate on the evaluation itself. That's the difference between evaluating on Tuesday and evaluating next quarter.

The connection is a URL, not an SDK. Streaming is a WebSocket at wss://streaming.assemblyai.com/v3/ws. The Voice Agent API is a WebSocket at wss://agents.assemblyai.com/v1/ws. Sync is an HTTP POST. You can hit all three with tools you already have — the Voice Agent API works with Claude Code out of the box with no SDK required — which means a spike takes an afternoon and doesn't commit you to a dependency.

Tuning is a parameter, not a project. Turn detection is exposed rather than hidden: min_turn_silence and max_turn_silence come from the mode preset you pick (128 ms and 1280 ms on balanced), vad_threshold defaults to 0.2, and ForceEndpoint lets your application end a turn when it knows something the model doesn't. Accuracy tuning is one field — pass agent_context with the question you just asked. Vocabulary is keyterms you can update mid-stream. None of these require a retrained model, a support ticket, or a custom deployment.

Switching costs are deliberately low. Drop-in plugins for LiveKit and Pipecat mean the speech layer is a config change. Migrating from another provider's streaming API is mostly a WebSocket URL and a parameter map — which is the same reason it's easy to migrate away, and we'd rather compete on that basis. Retell, for instance, integrates AssemblyAI as its high-accuracy speech-to-text option.

The failure modes are legible. The response field speech_model_used tells you which model actually ran on a pre-recorded job, so "the transcript got worse last Tuesday" is a question you can answer instead of a mystery. Partial transcripts arrive at roughly 750 ms for the first one and about every three seconds after, so you can build UI against a known cadence.

The pattern across all of that: nothing important is hidden behind a support conversation. That's the whole design philosophy, and it's why choosing a speech-to-text API for a voice agent should start with a two-hour spike rather than a vendor deck.

Running in production: support, scale, and security review

The demo is the easy part. Here's what the next eighteen months look like.

When something breaks at 2am. Start with the boring answer: the platform runs at 99.99% uptime across 600M+ inference calls a month, so the most common outcome is that nothing breaks. When something does, the first thing you want is to know whether it's you or us — there's a public status page for that, and it's the first URL to bookmark. Enterprise support agreements with defined response commitments are available, and support is scoped through the enterprise team rather than a ticket queue you shout into.

But the more useful question is how much of a 2am incident the architecture prevents. Session resume within 30 seconds means a network blip doesn't drop the conversation. LLM Gateway's automatic cross-provider fallback means a model provider's outage doesn't become your outage. Unlimited concurrency means a traffic spike doesn't produce refused connections at 2am, which is the classic pager-at-night failure on capped platforms. And a managed single API collapses the number of vendors you have to triage across from four to one — at 2am, "which layer is broken" is most of the incident.

At enterprise call volume. The platform handles 100,000 concurrent streams with no per-stream fee and no concurrency tier, serving 1,500 corporate customers and over a million developers. Billing is per second with no minimums, so a 10x volume spike costs 10x and nothing else — there's no committed tier to blow through and no overage rate. For high-call-volume contact centers, that combination of unlimited concurrency and linear cost is usually the deciding factor, and it's worth reading alongside the hidden costs of a voice agent stack.

Getting through procurement and security review. This is where voice agent projects die quietly, six weeks after the technical evaluation passed.

The compliance stack is SOC 2 Type 2, ISO 27001:2022, and PCI DSS v4.0, and the Voice Agent API is PCI-certified end to end — which matters if your agent will ever hear a card number. EU data residency runs on api.eu.assemblyai.com and streaming.eu.assemblyai.com at the same price as US endpoints, so residency is a configuration decision rather than a contract negotiation. Self-hosted deployment exists for teams whose security posture requires it.

For healthcare: AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI. The BAA can be signed in minutes without a sales call.

The structural advantage here is vendor count. A four-vendor assembled stack means four security reviews, four DPAs, four subprocessor lists, and four renewal cycles — and the review takes as long as the slowest vendor. Consolidating speech-to-text, LLM, and TTS behind one API doesn't make your security team less thorough. It gives them one thing to be thorough about.

Performance benchmarking and optimization strategies

Key performance metrics

On the first row: Universal-3.5 Pro Realtime's time-to-final is a 285 ms median, 374 ms at p95, and 443 ms at p99, with no word error rate cost from the speed. The p99 is the number to care about — it's the one your angriest caller experiences.

Optimization strategies

One more that isn't a technique so much as a discipline: give the model conversation context. Feeding real context into transcription is worth more than most latency tuning, and we've written up how conversation context changes voice agent accuracy with the numbers behind it.

Choosing the right architecture for your voice AI agent

Short version, by situation.

Prototyping, or the agent isn't on a live call? Cascading pipeline. Ship it, learn from it, don't over-engineer.

Your app owns turn detection — push-to-talk, dictation, IVR? Sync API. One POST per utterance, no session state, ~134 ms p50.

You need per-component control and have the engineers? Streaming pipeline on LiveKit Agents or Pipecat, with Universal-3.5 Pro Realtime underneath. Maximum flexibility, maximum surface area.

You need a production phone agent and time-to-launch matters? Managed single API. One socket, one bill, one security review, roughly 1 second end-to-end.

Migrating from an assembled stack? Move the speech layer first. It's the component with the clearest measurable improvement, it's a config change on LiveKit or Pipecat, and it lets you validate accuracy gains on production traffic before you commit to consolidating the rest. Then collapse LLM and TTS once you trust the numbers. Teams that try to swap all three layers in one release ship six weeks late and can't attribute the result. There's a fuller treatment in our comparison of voice agent architectures.

Final words

The uncomfortable thing about the last two years is that the stack got easier to build and no easier to get right. Public trust in AI systems remains a genuine constraint — Stanford HAI's AI Index tracks a public that is markedly more ambivalent than the deployment curve suggests — and every clumsy phone agent spends a little of everyone's credit.

Here's the thing worth taking away, though, and it's not "pick the fastest model."

Every architecture decision in this post is really a decision about how many seams you're willing to own. Seams are where latency accumulates, where errors compound, where incidents get hard to diagnose at 2am, and where security reviews multiply. Component choice is a Tuesday afternoon. Seam count is the thing you live with for years — and it's the variable most teams never explicitly choose, because it gets decided implicitly by whichever tutorial they started from.

Choose it on purpose.

Frequently asked questions

Which voice agent platform has the best developer experience?

The best developer experience is the one where you can get audio flowing before you talk to a salesperson, and tune behavior with a parameter instead of a support ticket. AssemblyAI's Voice Agent API is a single WebSocket at wss://agents.assemblyai.com/v1/ws that works with Claude Code out of the box with no SDK required, and the free tier — 185 hours pre-recorded plus 333 hours streaming — needs no contract or minimum commitment to start. Turn detection is exposed as configuration (min_turn_silence and max_turn_silence set by the mode preset, 128 ms and 1280 ms on balanced, vad_threshold at 0.2, plus ForceEndpoint), accuracy tuning is one field (agent_context), and keyterms update mid-stream via UpdateConfiguration. Drop-in plugins for LiveKit and Pipecat make the speech layer a config change rather than a refactor.

What support is available if my voice agent has issues in production at 2am?

A public status page tells you within seconds whether the problem is yours or ours, and enterprise support agreements with defined response commitments are available through the enterprise team. Most of the answer, though, is architectural: the platform runs 99.99% uptime across 600M+ inference calls a month, sessions resume within 30 seconds of a disconnect using session.resume and the prior session_id, LLM Gateway fails over automatically across providers, and unlimited concurrency means a traffic spike produces a bigger bill rather than refused connections. A single managed API also collapses incident triage from four vendors to one, which at 2am is most of the work.

Best voice agent API for enterprise-scale, high-call-volume production

For high-call-volume production, look for unlimited concurrency, per-second billing, and no committed-volume tier — AssemblyAI's Voice Agent API has all three at a flat $4.50/hr, and the platform sustains 100,000 concurrent streams serving 1,500 corporate customers. There are no per-stream fees, per-agent subscriptions, or concurrency ceilings, so a 10x volume spike costs 10x with no overage rate and no refused calls. Accuracy holds at scale too: 6.99% pooled word error rate and 15.31% entity error rate on Pipecat's open benchmark of real agent conversations, with time-to-final at a 285 ms median and 443 ms at p99.

Which voice agent API is easiest to get approved by procurement and security teams?

The easiest one to approve is the one that replaces four vendor reviews with one. AssemblyAI carries SOC 2 Type 2, ISO 27001:2022, and PCI DSS v4.0, and the Voice Agent API is PCI-certified end to end, which matters for any agent that will hear a card number. EU data residency runs on api.eu.assemblyai.com and streaming.eu.assemblyai.com at the same price as US endpoints, so residency is a configuration choice rather than a contract negotiation, and self-hosted deployment is available where security posture requires it. For PHI workloads, AssemblyAI is considered a business associate under HIPAA and offers a standard Business Associate Addendum that can be signed in minutes without a sales call.

Single voice agent API vs building your own STT-LLM-TTS stack: which is better?

Build your own when you need per-component control and have engineers to spend on the integration surface; use a single API when time-to-launch, predictable cost, and a smaller incident and security footprint matter more. An assembled stack on LiveKit Agents or Pipecat gives you maximum flexibility and gets you under 1,500 ms end-to-end with work, but you own every vendor upgrade, every turn-taking edge case, and four security reviews. AssemblyAI's Voice Agent API bundles speech-to-text, LLM, and TTS behind one WebSocket at a flat $4.50/hr, lands around 1 second end-to-end, and includes turn detection, interruption handling, Voice Focus, recordings, and hosting in the base rate.

What's the migration path from a self-built STT-LLM-TTS stack to a single Voice Agent API?

Move the speech-to-text layer first, then collapse LLM and TTS once the accuracy gain is proven on production traffic. Speech is the right first step because it has the clearest measurable improvement, and on LiveKit or Pipecat it's a config change through AssemblyAI's drop-in plugins rather than a refactor — which lets you A/B two speech models on real calls without shipping code. Measure entity error rate, not just word error rate, during that phase; entity accuracy is where the downstream cost lives. Teams that swap all three layers in one release ship late and can't attribute the result to any single change.

‍