The voice AI stack for building agents in 2026

Most voice agent projects don't fail on the model. They fail in the seams between models.

You pick a good speech-to-text API, a good LLM, a good TTS voice, wire them together, and the demo is charming. Then you put it on a real phone line with a real caller who has an accent, a barking dog, and an account number to read out, and the whole thing falls apart—not because any single component is bad, but because each one adds a few hundred milliseconds and a few percent of error, and those compound.

McKinsey's "The State of AI" put 62% of organizations somewhere on the AI agent curve—experimenting or scaling. A lot of those teams are about to learn the same lesson: the stack is the product. So here's what the stack actually looks like in 2026, layer by layer, with the numbers that matter and the tradeoffs nobody puts in the marketing copy.

Deconstructing the modern voice agent stack

Four components, and they have to work as one system.

Speech-to-text is the ears. It converts the caller's audio into text your agent can reason over. Everything downstream inherits its mistakes.

The LLM is the brain. It decides what to say and which tools to call.

Text-to-speech is the voice. It turns the response back into audio.

Orchestration is the conductor. It manages the WebSocket connections, the turn-taking, the interruptions, the state, and the dozen race conditions that show up the first time somebody talks over your agent.

A single conversational turn runs through all four in sequence: caller speaks → STT transcribes → LLM reasons → TTS synthesizes → audio plays back. The math is unforgiving. If each stage takes 400ms, you're at 1.6 seconds before the caller hears anything, and 1.5 seconds is roughly where a conversation stops feeling like a conversation.

That's the whole game. Not "which model is best" but "where does the time go, and what can overlap."

Voice AI agent architecture

Picture the four components arranged around an orchestration layer that owns every connection.

Inbound audio streams from the caller into STT over a WebSocket. Partial transcripts come back continuously; a final transcript arrives when the model decides the turn is over. The orchestrator hands that final turn to the LLM, streams the first tokens into TTS as they arrive, and pushes synthesized audio back to the caller—often while the LLM is still generating the rest of the sentence.

Each component has its own optimization target. STT optimizes for accurate finals fast. The LLM optimizes for time-to-first-token, not total throughput. TTS optimizes for time-to-first-audio-byte. And the orchestrator optimizes for never blocking.

The conductor metaphor holds up because the hard part isn't the players. It's the timing.

Voice AI architecture patterns

Three ways to assemble this, with real tradeoffs.

Cascading pipelines

Each component runs to completion before the next one starts. The caller finishes speaking, STT returns a full transcript, the LLM generates a full response, TTS synthesizes the whole thing, then audio plays.

Simple to build and simple to debug. Also 800–2000ms of latency, which rules it out for live conversation. Cascading is fine for asynchronous work—call summarization, post-call analytics, anything where nobody's waiting—and it's a reasonable way to prototype logic before you optimize.

Streaming architectures

Components overlap. STT emits partials while the caller is still talking. The LLM starts generating on the final transcript and streams tokens out. TTS synthesizes in chunks and starts playing the first chunk before the last token exists.

This is what production voice agents run on, and it's the only pattern that handles barge-in properly—because to let a caller interrupt, you need to be able to stop mid-sentence, flush the audio buffer, and re-enter listening state without tearing down connections.

Universal-3.5 Pro Realtime is built for this pattern. It runs over wss://streaming.assemblyai.com/v3/ws, returns partials continuously, and detects end-of-turn at roughly 300ms using the punctuation it predicts rather than a silence timer. That distinction matters more than it sounds like it should: a silence-based endpointer cuts off a caller who pauses mid-way through reading a credit card number, and a caller who gets cut off mid-number is a caller who hangs up.

End-to-end models

Single models that take audio in and emit audio out, skipping the text layer entirely. Genuinely impressive, and improving fast.

They're still hard to run in production for most teams, because you give up the things you actually need at scale: a text transcript to log and audit, the ability to swap the LLM without changing your speech vendor, tool-calling reliability, and per-component observability when something breaks at 2am. Modular streaming is still the right default for anything with a compliance requirement or a support rotation.

Speech-to-text: the foundation

STT is the entry point, and its errors are the only errors in the stack that nothing downstream can recover from. A hallucinated word becomes a wrong LLM decision becomes a confidently wrong spoken answer. The caller doesn't hear "transcription error." They hear an agent that isn't listening.

Word error rate is the standard starting metric and it's a genuinely incomplete one—WER treats every word as equally important, which is wrong for voice agents. Getting "um" wrong costs nothing. Getting a phone number wrong costs the call. That's why entity error rate is the number to watch, and it's the number most vendors don't publish.

Here's Universal-3.5 Pro Realtime on Pipecat's open STT benchmark, which uses real agent conversations rather than clean read speech. Lower is better.

Look at the entity column against the WER column. Google Chirp3 is within about two points of us on pooled WER and 40% worse on entities. That gap is the difference between a benchmark win and a working agent.

Three capabilities in Universal-3.5 Pro Realtime exist specifically because of how voice agents fail:

agent_context. Pass your agent's own question into the session and the model knows what kind of answer is coming. After "what's your email address?" the reply resolves to user@assemblyai.com instead of "user at assemblyai dot com." Across a benchmark of 20,000 voice agent audio files, passing agent context cut WER by 10.2%. Context Carryover—a rolling memory of prior finalized turns—is on by default, so the user half of the conversation comes along for free.

voice_focus. Speaker isolation with two variants: near-field for headsets and handsets, far-field for rooms, kiosks, and drive-thrus. It suppresses background speech before the audio ever reaches the transcription model.

mode. Instead of a dozen low-level flags, three presets: min_latency, balanced (the default, and the right one for interactive agents), and max_accuracy for noisy or far-field audio.

Setting it up is a connection-parameter change, not an architecture change:

CONNECTION_PARAMS = {
    "sample_rate": 16000,
    "speech_model": "universal-3-5-pro",
    "mode": "balanced",
    "voice_focus": "near-field",
    "agent_context": "Thanks for calling—what can I help you with today?",
}

Set speech_model explicitly. A config that omits it rides whatever the account default happens to be, which is exactly the kind of thing that quietly changes underneath you.

The rest of the sheet: 18 languages with mid-sentence code-switching (Hinglish included), live speaker diarization that labels speakers as they talk and sends a single re-clustering correction within about half a second of the stream ending, and keyterm prompting included at no extra cost.

Pricing is $0.45/hr base, billed on session duration, with add-ons that stack only if you use them: diarization with revision +$0.12/hr, prompting +$0.05/hr, voice isolation +$0.10/hr. agent_context, rolling memory, and keyterm prompting are included. Concurrency is unlimited, and new-session rate limits scale automatically—which matters more than the hourly rate the first time a marketing campaign triples your call volume on a Tuesday.

If you need coverage beyond those 18 languages, Universal-2 handles 99+ at $0.15/hr. It's the right call for long-tail language support and legacy integrations, not for a latency-sensitive agent.

Text-to-speech: giving voice to AI agents

Pew Research found in March 2022 that 47% of people were concerned about AI handling customer service calls. Four years on, the concern has mostly migrated from "will it understand me" to "will it sound like a robot reading a script"—and TTS is where you win or lose that.

The metrics that matter:

• Time to first audio byte. Under 200ms. This is the number that determines whether your agent feels responsive, because it's the last thing the caller waits on.
• Mean opinion score. Above 4.0 for genuinely human-like quality.
• Prosody and emotional range. Whether the voice can ask a question that sounds like a question.
• Streaming chunk support. Non-negotiable. If your TTS won't start speaking before the full text arrives, you've reintroduced a cascading pipeline into an otherwise streaming stack.

Cartesia, ElevenLabs, and Rime all ship production-grade streaming voices with different strengths on latency, voice cloning, and pricing. We keep a running breakdown in our text-to-speech API comparison.

LLMs: the brain of voice AI agents

The model you'd pick for a chat product is usually the wrong one for voice.

Voice inverts the priorities. In text, total throughput and response quality dominate. In voice, time-to-first-token is nearly everything, because the caller is sitting in silence until the first audio byte lands, and the first audio byte can't be synthesized until the first token exists.

What to optimize for:

• Time-to-first-token over total generation speed. A model that starts in 200ms and generates slowly beats one that starts in 600ms and sprints.
• Brevity under instruction. Voice responses should be one or two sentences. Most models over-explain by default and you'll fight this in the system prompt.
• Reliable function calling. Voice agents live or die on tool calls—looking up an order, checking availability, transferring to a human.
• Context handling across turns. The transcript grows every turn, and so does the latency if you're not managing the window.

Frontier models like Gemini 3 Pro and GPT-5.2 are the right choice when the agent has to reason over a complex policy or a long document. For the latency-critical path, smaller models like Gemini 2.5 Flash-Lite or Claude 4.5 Haiku are usually the better trade—the inference cost for capable small models has dropped sharply, per Stanford HAI's AI Index.

The pattern worth stealing: tier your models. Run the fast one for the conversational path, and escalate to the frontier model only for the turns that need it. If you're calling several providers, the LLM Gateway gives you one API across OpenAI, Anthropic, and Google so switching is a parameter change rather than a refactor.

Orchestration

This is the layer people underestimate, and it's where most of the engineering time actually goes.

Orchestration isn't passing data between components. It's managing concurrent WebSocket connections that can each fail independently, deciding when a turn has ended, killing TTS playback mid-word when the caller interrupts, tracking conversation state across tool calls, handling reconnects without dropping the call, and logging enough to debug all of it later.

You've got two paths.

Build on a framework

Vapi, LiveKit Agents, and Daily/Pipecat each give you the plumbing for building AI voice agents—audio transport, turn-taking, interruption handling, plugin interfaces for STT/LLM/TTS—while leaving the component choices to you. AssemblyAI ships drop-in plugins for LiveKit and Pipecat, so swapping the STT layer is a config change.

This is the right path when you care about component-level control: picking a specific voice, a specific model, a specific endpointing profile, and being able to change any one of them without changing the others.

We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover—it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms.

— David Zhao, Co-founder at LiveKit

Use a managed voice agent API

The other path is to stop assembling the pipeline yourself. AssemblyAI's Voice Agent API is one WebSocket—wss://agents.assemblyai.com/v1/ws—that bundles STT, LLM, and TTS behind a single connection at a flat $4.50/hr. One bill, one set of logs, one thing to reconnect. It runs on Universal-3.5 Pro Realtime for speech, lands around 1 second end-to-end, and works with Claude Code out of the box with no SDK required.

That's not the same product category as Bland, Retell, or Synthflow, and it's worth being precise about the difference. Those are agent platforms—they own the dashboard, the flow builder, and increasingly the customer relationship. The Voice Agent API is infrastructure. It doesn't have opinions about your conversation design; it gives you the pipeline as a primitive and gets out of the way. We wrote up the reasoning behind that in how to build with the Voice Agent API.

Whichever path you take, the capabilities you're looking for are the same: streaming connection management, turn-taking and interruption handling, conversation state, function calling into your systems, and observability you can actually debug with.

Performance benchmarking and optimization strategies

You can't optimize what you haven't instrumented. Log a timestamp at every boundary: caller stops speaking, STT emits the final turn, LLM returns its first token, TTS returns its first audio byte, audio starts playing. Then find the biggest number and attack that one.

Key performance metrics

The entity row is the addition worth arguing about. If your agent takes orders, books appointments, or verifies identity, entity accuracy predicts task completion far better than pooled WER does. Our guide on how to evaluate speech recognition models walks through building a test set from your own calls rather than trusting anyone's published numbers, including ours.

Optimization strategies

Two more that don't fit neatly in a table. First, tune your endpointing to the call type—rapid order confirmations want aggressive settings, while healthcare and legal conversations need conservative ones so a thinking pause doesn't end the turn. Second, measure concurrency under real load before you launch, not after. Concurrency limits tend to reveal themselves at the worst possible moment.

Choosing the right architecture for your voice AI agent

A rough decision rule:

• Prototyping the conversation logic? Cascading. Get the flow right, then optimize.
• Shipping a production agent where you need to control the voice, the model, and the endpointing independently? Streaming pipeline on LiveKit Agents or Pipecat, with Universal-3.5 Pro Realtime underneath.
• Shipping soon with a small team, and STT/LLM/TTS integration isn't your differentiator? Managed API. One connection at $4.50/hr, and you keep the ability to move.
• The owner isn't an engineering team? An agent platform, with clear eyes about the lock-in.

The mistake to avoid is picking based on today's requirement and discovering the ceiling six months in. We wrote about exactly where those ceilings show up in the production ceiling.

Final words

The interesting shift since January isn't that the components got faster. It's that the boundaries between them started leaking—on purpose.

agent_context is the clearest example. It's an STT parameter whose entire value comes from knowing what the LLM just said. The speech model gets 10.2% more accurate not because the acoustics improved but because it stopped being treated as a component with a clean interface. Same story with transcript-based endpointing, which is a turn-taking decision made inside the speech model rather than by the orchestrator, and with tiered LLM routing, which is a latency decision that lives above the model.

The clean four-box diagram at the top of this post is a useful teaching tool and an increasingly bad architectural target. The stacks that feel natural in 2026 are the ones where information flows sideways between layers, not just forward through them. If your components are perfectly encapsulated, you're leaving accuracy and latency on the table.

So when you evaluate a piece of this stack, don't just ask how good it is in isolation. Ask what it can be told about the rest of the conversation—and what it does with that.

Frequently asked questions

What's the difference between a voice AI stack and a conversational AI platform?

A voice AI stack is the modular set of components you assemble yourself—speech-to-text, an LLM, text-to-speech, and orchestration—with each piece chosen and swapped independently. A conversational AI platform bundles all of it behind a dashboard and a flow builder, which is faster to launch and harder to move off. There's a middle option too: a managed API like AssemblyAI's Voice Agent API gives you the whole pipeline through one WebSocket at a flat $4.50/hr without handing over your conversation design.

How do I measure the latency of my voice AI agent?

Log a timestamp at each boundary in the turn: when the caller stops speaking, when the speech-to-text final transcript arrives, when the LLM returns its first token, and when text-to-speech returns its first audio byte. The gaps between those timestamps tell you which component owns your latency, and it's usually not the one you assumed. Aim for under 1500ms total from end-of-speech to first audio, with time to first byte under 200ms at each stage.

What speech-to-text model should I use for a voice agent?

Universal-3.5 Pro Realtime (universal-3-5-pro) is the current flagship for real-time voice agents, at $0.45/hr base over wss://streaming.assemblyai.com/v3/ws. It scores 6.99% pooled word error rate and 15.31% entity error rate on Pipecat's open STT benchmark, supports 18 languages with mid-sentence code-switching, and accepts agent_context so the model knows what your agent just asked. For 99+ language coverage where latency is less critical, Universal-2 runs at $0.15/hr.

Can I mix and match providers for each component of the stack?

Yes, and most production teams do. A modular streaming architecture on LiveKit Agents or Pipecat lets you pick the best speech model, the best LLM for your latency budget, and the voice your brand wants, then change any one of them without touching the others. The tradeoff is that you own the integration surface—four vendors means four sets of credentials, four failure modes, and four bills.

How much does streaming speech-to-text cost for a voice agent?

Universal-3.5 Pro Realtime is $0.45/hr base, billed on session duration with no minimums, which works out to about three-quarters of a cent per minute of connected call time. Add-ons stack only if you use them: diarization with revision +$0.12/hr, prompting +$0.05/hr, voice isolation +$0.10/hr, while agent_context, rolling conversation memory, and keyterm prompting are included. Full pricing is on the pricing page.

When should I build a component in-house instead of using an API?

Building a foundational speech or language model needs a dedicated research team, a data pipeline, and a multi-year budget—it's a company, not a feature. Almost every team is better off using production APIs for the model layer and spending their engineering time on the orchestration, the conversation design, and the integrations into their own systems, which is where the actual product differentiation lives. The exception is a genuine data or deployment constraint, like fully air-gapped audio, which is what self-hosted deployments are for.

‍