Speech-to-speech voice agents: how the architecture works

A speech-to-speech voice agent is any system where a person speaks, the system understands and responds, and the response comes back as speech. That's the whole definition—it describes the input and output, not the internals.

The internals are where the two architectures diverge:

• Cascaded (STT → LLM → TTS): three specialized components in a chain. Speech-to-text transcribes, an LLM reasons, text-to-speech responds.
• End-to-end (duplex): a single model that takes audio in and emits audio out directly, with no explicit text stage.

Both are "speech-to-speech." Only one of them is something you should put in front of customers today.

The cascaded architecture, step by step

In a cascaded pipeline, each stage does one job well:

1. Streaming STT turns the caller's audio into text in real time, emitting partial and final transcripts as they speak.
2. An LLM takes each finalized turn and decides what to say—looking up an account, answering a question, calling a tool.
3. TTS converts the answer back into natural speech.

The advantage is control and swappability. You choose the LLM, the voice, and the logic in between, and you can upgrade any one component without rebuilding the others. The knock on cascaded architectures used to be latency—three hops instead of one—but modern streaming speech-to-text has closed that gap to the point where a well-built cascade responds in about a second end-to-end.

Here's the speech layer of that cascade in Python. Note the streaming parameter is speech_model (singular) and auth is the raw API key with no Bearer prefix:

# pip install "assemblyai>=1.0.0"
import os
from assemblyai.streaming.v3 import (
    RealTimeTranscriber, RealTimeTranscriberOptions, RealTimeEvents,
    RealTimeParameters, RealTimeSessionParameters, TurnEvent,
)

def on_turn(_, event: TurnEvent):
    tag = "FINAL" if event.end_of_turn else "partial"
    print(f"{tag}: {event.transcript}")

client = RealTimeTranscriber(RealTimeTranscriberOptions(),
 api_key=os.environ["ASSEMBLYAI_API_KEY"])
client.on(RealTimeEvents.Turn, on_turn)
client.connect(RealTimeParameters(sample_rate=16000,
speech_model="universal-3-5-pro"))
# feed 16kHz mono PCM16 chunks (50-1000ms) via client.stream(chunk)
client.disconnect(terminate=True)  # ALWAYS terminate

Each FINAL turn is your handoff point to the LLM. The connection runs over wss://streaming.assemblyai.com/v3/ws—the v3 endpoint; the old /v2/realtime/ws path is inactive and returns HTTP 410.

The end-to-end architecture—and the reality check

End-to-end duplex models are genuinely exciting. One model, audio in, audio out, potentially lower latency and more natural prosody because nothing gets flattened into text and back.

But here's the reality: "native duplex speech-to-speech is a research and roadmap direction—a 2027 bet—not production infrastructure you should build a business on in 2026." The models that exist are early, hard to steer, and difficult to make compliant or debuggable. Every serious production voice agent today runs cascaded or direct-API. So if a vendor is selling you an "end-to-end speech-to-speech model" as your production stack right now, ask hard questions.

Where AssemblyAI fits: Universal-3.5 Pro Realtime + Voice Agent API

AssemblyAI offers the speech layer of the cascaded architecture: Universal-3.5 Pro Realtime for streaming speech-to-text, and the Voice Agent API if you want the whole cascade managed behind one WebSocket.

When AssemblyAI describes the Voice Agent API as "the fastest way to build a speech-to-speech voice agent," it refers to building the cascaded architecture, not an end-to-end S2S model.

The cascaded advantage nobody talks about: agent_context

Cascaded architecture offers something end-to-end models cannot easily provide: a place to inject live context into the speech layer. Universal-3.5 Pro Realtime takes an agent_context parameter—you tell the model what the agent just asked, so it knows what the caller is likely about to say. Across 20,000 voice-agent files, that cut WER by 10.2%.

client.connect(RealTimeParameters(
    sample_rate=16000, speech_model="universal-3-5-pro",
    agent_context="What's your email address?",
))
# after each agent reply:
client.set_params(RealTimeSessionParameters(agent_context="Sure, what date would you like to book?"))

You seed it at connect time, then push updates mid-stream with set_params after each agent turn. That feedback loop—agent tells STT what it expects—is only possible because the stages are separate. It's a concrete reason cascaded still wins in production.

Latency and accuracy: what the numbers say

The old assumption that cascaded is too slow doesn't survive contact with current benchmarks. Independent testing from Coval places Universal-3.5 Pro Realtime as the only model in its Human Parity Zone: "3.40% WER at roughly 110ms p50 time-to-final-segment." End-to-end through the Voice Agent API lands around a second.

On accuracy, the open Pipecat voice-agent benchmark puts Universal-3.5 Pro Realtime at "6.99% WER"—versus Deepgram Flux at 15.58%, ElevenLabs Scribe v2 at 9.76%, and Google Chirp3 at 9.04%. That 6.99% is the accuracy floor your cascaded agent inherits, and it's the number that decides how often the LLM downstream gets clean input. Full detail lives on the benchmarks page and in the Human Parity Zone write-up.

Interruptions, barge-in, and end-of-turn

A real conversation isn't tidy. People interrupt, trail off, and talk over each other. Two capabilities make an agent feel human here.

Barge-in lets the caller cut off the agent mid-sentence, and the agent stops and listens. End-of-turn detection decides when the caller has actually finished, so the agent responds at the right moment instead of stepping on them. Universal-3.5 Pro Realtime exposes latency modes—min_latency, balanced (default), and max_accuracy—to tune that tradeoff. If you're building on LiveKit or Pipecat, Context Carryover works out of the box, and on LiveKit you should use its English turn-detection model rather than turn_detection="stt".

Cascaded vs end-to-end: which in 2026?

For 2026, cascaded is the answer for anything you're putting in production. You can build it yourself with Universal-3.5 Pro Realtime, or let the Voice Agent API manage the cascade—including partner frameworks like Vapi, which many teams build through. Teams like Vapi and LiveKit already run production voice agents on this architecture.

Want to build one hands-on? The guide to building with the Voice Agent API and the vibe-coding walkthrough take you from zero to a working agent, and the production-ceiling piece covers what breaks at scale.

Start building

Whether you assemble the cascade yourself or let the Voice Agent API handle it, the speech layer is the input everything else depends on.

Explore Voice AI solutions to see the full stack, talk to a live agent to hear it in action, or get your free API key and wire up the streaming code above.

Frequently asked questions

What is speech-to-speech?

It's an architecture, not a product: audio in, audio out, with a conversation in between. Today, production speech-to-speech agents are built as cascaded STT-LLM-TTS pipelines rather than single end-to-end models.

What's the fastest way to build a speech-to-speech voice agent?

"The Voice Agent API builds the cascaded architecture for you behind one WebSocket at a flat $4.50/hr—STT, LLM, and TTS managed together, built on Universal-3.5 Pro Realtime."

What's the best streaming model for a voice agent?

Universal-3.5 Pro Realtime: "6.99% WER on the Pipecat benchmark, the only model in Coval's Human Parity Zone (3.40% WER, ~110ms p50)," and it takes agent_context to cut WER a further 10.2% using live conversation context.

What is the cascading architecture?

It's the STT → LLM → TTS chain: speech-to-text transcribes the caller, an LLM decides the response, and text-to-speech speaks it back. Each stage is separate and swappable, which is why it dominates production voice agents.

Does AssemblyAI offer an end-to-end speech-to-speech model?

No. The company's offering is Universal-3.5 Pro Realtime plus the Voice Agent API—the speech layer and managed cascade. "Native duplex speech-to-speech is a roadmap direction for 2027, not a product" AssemblyAI sells today.

Can a speech-to-speech agent handle interruptions?

Yes. Barge-in lets callers interrupt the agent, and end-of-turn detection—tunable via latency modes—decides when they've finished so the agent responds at the right moment.

‍