Best API for building a speech-to-speech voice agent in 2026

A developer's guide to the voice agent APIs available in 2026—what each one is best at, how they compare on speech accuracy, latency, and pricing, and how to choose between a single-API approach and a chained STT-LLM-TTS pipeline.

A voice agent API replaces the three separate components most teams used to wire together—streaming speech-to-text, a language model, and text-to-speech—with a single API that takes audio in and returns audio out. In 2026, that category has gone from "interesting demo" to "default way to ship a production voice agent," and the gap between providers is now measurable in latency, accuracy, and what they let you do with tool calls.

This guide compares the voice agent APIs developers actually pick from in 2026, what each one is best at, and how to choose between a true speech-to-speech pipeline and chaining STT-LLM-TTS yourself. We'll cover AssemblyAI's Voice Agent API, OpenAI Realtime, ElevenLabs Conversational AI, Retell, and Vapi, plus a raw DIY stack, and where AI voice agents land depending on how much of the pipeline you want to own—covered in more depth in our orchestration tools comparison.

What is a speech-to-speech voice agent?

A speech-to-speech voice agent is a voice agent exposed behind a single API endpoint—usually a WebSocket—that accepts a user's audio stream and returns the agent's audio response, with everything in between (transcription, reasoning, tool calls, voice synthesis) hidden behind one connection. You send mic audio in. You get the agent's voice back. You don't manage three providers, three sets of API keys, or three sets of latency budgets.

That's the practical definition. Under the hood, there are two architectural patterns:

• Chained (cascading) speech-to-speech pipelines: Internally pipe streaming STT into an LLM into streaming TTS, but expose a single API. The advantage is you can swap each layer for a best-in-class model and inspect the text at every hop. AssemblyAI's Voice Agent API is the leading example.
• Native speech-to-speech models: A single model trained end-to-end on audio that takes audio tokens in and emits audio tokens out, with no intermediate text in some cases. OpenAI Realtime falls here. The pitch is lower latency and richer audio understanding (laughter, tone). The trade-off is less transparency, smaller language support, and weaker text reasoning than a frontier text LLM.

Both expose the same developer surface—one connection, audio in, audio out—so the choice is about which trade-offs match your application.

Best voice agent APIs in 2026

Here's how the main options rank on the things that decide whether a production voice agent ships. This table is current as of July 2026.

A few things stand out in 2026. End-to-end latency under one second is now table stakes, not a differentiator—every provider on this list will get you there with a reasonable network. What separates them is speech accuracy on real-world audio (phone calls, accents, alphanumerics), how tool calling behaves under load, and whether the pricing model survives contact with a real customer base.

It's also worth being precise about what "API" means here. Vapi and Retell are platforms—you build inside their environment and accept their orchestration, billing, and lock-in. AssemblyAI's Voice Agent API and OpenAI Realtime are APIs—you call an endpoint from your own stack. If you want to own your architecture and avoid platform lock-in, that distinction matters more than any single benchmark row.

In our Voice Agent Report, 76% of respondents rated speech-to-text accuracy as the single most important non-negotiable when building voice agents—above latency, cost, and integration capabilities. That maps directly to the comparison data: the accuracy gap between providers on real-world entities (phone numbers, emails, confirmation codes) is where production agents succeed or fail.

What changed in 2026

The biggest shift this year is that the speech foundation under the best voice agent APIs got materially more accurate—and multilingual. As of mid-2026, AssemblyAI's Voice Agent API runs on Universal-3.5 Pro Realtime, the new flagship realtime model. Three things matter for voice agents specifically:

• Accuracy on real conversations, not studio clips. On the Pipecat open STT benchmark—built from real agent conversations—Universal-3.5 Pro Realtime posts a pooled word error rate of 6.99%, ahead of Deepgram Flux (15.58%), ElevenLabs Scribe v2 (9.76%), and Google Chirp3 (9.04%). Entity error rate is 15.31% versus 50.50% for Deepgram Flux and 39.70% for ElevenLabs Scribe v2.
• Context carryover. Rolling conversation memory is on by default, and you can pass the agent's own question as agent_context so the model hears the user's reply through the lens of what was just asked. Across 20,000 voice-agent files, passing agent context cut WER by 10.2%, with fabrications down 18.3% and hallucinations down 17.2%. One team paired agent context with prompting and dropped utterance error rate from 26% to 9% on production audio.
• 18 languages and ~300ms turn detection. The model covers 18 languages (up from 6) with mid-sentence code-switching, and its end-of-turn detection reads tonality, pacing, and rhythm rather than a fixed silence timer—landing around 300ms.

The practical upshot: the "which voice agent API is most accurate" question got easier to answer in 2026, and the accuracy edge now holds across more languages and messier audio than it did a year ago.

How to choose the best voice agent API

The voice agent you ship depends on four decisions. Get any of them wrong and the agent feels off, even if the demo was great.

1. Speech-to-text accuracy on your actual audio

Most providers benchmark on studio audio. Your users are on phones, in cars, in drive-thrus, and rattling off order numbers and email addresses. Two metrics actually matter:

• Alphanumeric accuracy: How well the model captures phone numbers, confirmation codes, emails, and order IDs. This is where the gap between providers shows up most clearly. On agent-conversation audio, Universal-3.5 Pro Realtime posts a 15.31% entity error rate—well below Deepgram Flux (50.50%) and ElevenLabs Scribe v2 (39.70%)—including 3.55% on phone numbers versus 10.41% for Deepgram Flux. That's the difference between capturing "RX-7704132" correctly on the first try and hearing "dash seven seven zero four one three two." This is the single most under-measured metric in voice agent demos.
• Entity accuracy on proper nouns: Company names, people's names, drug names, product titles. If your agent writes "Corel" instead of "Coral" into the CRM, the lead is unreachable.

Native speech-to-speech models like OpenAI Realtime were trained more on clean conversational audio than on telephony, which shows up the moment you put them on a Twilio call. If you're evaluating raw STT for a chained build, our best streaming speech-to-text for voice agents breakdown covers the same criteria.

2. Turn-taking and interruption handling

Poor turn detection is the most common reason voice agents feel unnatural. The agent either talks over the user or sits in awkward silence. The best implementations handle turn detection at the model level, not as an afterthought bolted on with a fixed silence timer.

Universal-3.5 Pro Realtime includes end-of-turn detection built directly into the model—reading tonality, pacing, and rhythm to land around 300ms rather than waiting on a fixed VAD timeout. Retell ships its own proprietary turn-taking model as a platform feature. OpenAI Realtime's server VAD is competent but configurable timeouts still trip up agents on calls with hesitant speakers.

3. Tool calling reliability

Real voice agents don't just talk—they book the appointment, look up the order, charge the card. That means the underlying LLM has to call tools mid-conversation, fast enough that the silence doesn't become obvious.

The bar to clear: tool calls under 500ms round-trip, structured outputs that don't hallucinate parameters, and the ability to call multiple tools in a single turn. But there's a UX dimension most teams overlook: what happens while the tool call is executing? AssemblyAI's Voice Agent API generates intermediate speech during tool execution—the agent says something like "Let me look that up for you" rather than going silent. OpenAI Realtime goes silent during tool calls, which creates an awkward dead-air gap that makes users wonder if the connection dropped.

If your agent's job is "capture data and put it somewhere"—booking a meeting, qualifying a lead, taking an order, scheduling a callback—tool calling reliability is what decides whether the agent actually does its job.

4. Pricing model and unit economics

This is the trap most teams fall into during pilots. Per-minute pricing looks cheap until you're running 500 simultaneous calls during a support spike. Per-token audio pricing (OpenAI Realtime) is unpredictable because audio output tokens cost far more than text tokens and a chatty TTS voice burns through your budget.

• Flat hourly pricing: AssemblyAI's Voice Agent API at $4.50/hour covers STT, LLM inference, TTS, and tool calling. One bill, one line of math to model what a 5-minute call costs. No separate meters for audio in, audio out, text in, text out. No concurrency commitments. Easy to forecast.
• Per-minute, all-in: Retell, ElevenLabs Conversational AI. Predictable, but adds up at scale—and ElevenLabs prices higher and caps concurrency around 30.
• Per-token audio: OpenAI Realtime. Roughly $18/hour with 30+ billing event types—about 4x the cost of the AssemblyAI Voice Agent API, and hard to forecast at scale.
• Pass-through + platform fee: Vapi. You pay each underlying provider plus a platform fee—flexible but more accounting overhead.

Forecast what 100 hours of conversation actually costs across the providers you're considering. The order of magnitude is real, especially once you stop being charged for demo calls and start being charged for production. When in doubt, check current pricing directly.

AssemblyAI Voice Agent API: one WebSocket, flat-rate, built on Universal-3.5 Pro Realtime

AssemblyAI's Voice Agent API is a single WebSocket that takes user audio in and streams agent audio out, with STT, LLM, TTS, turn detection, and tool calling handled inside the connection. It replaces three separate providers with one bill, one set of logs, and one set of latency variables to tune.

What makes it work:

• Speech accuracy that survives phone audio. The STT layer is Universal-3.5 Pro Realtime—6.99% pooled WER on real agent conversations, a 15.31% entity error rate, native 8kHz mulaw support, context carryover, and ~300ms turn detection. When the STT is this accurate, the whole conversation is better because the agent is actually responding to what was said.
• Tool calling that doesn't go silent. Define your tools, the model calls them, results stream back into the conversation. Unlike OpenAI Realtime, the agent generates intermediate speech during tool execution—natural transition phrases like "Let me check on that"—so your users never hear dead air.
• Context carryover on by default. Rolling conversation memory needs no configuration, and passing agent_context cut WER by 10.2% across 20,000 voice-agent files—fewer hallucinations, better entity capture on the replies that matter.
• Mid-session updates without reconnecting. Update the system prompt, voice, tools, and VAD settings mid-conversation with a JSON message—no reconnection, no redeployment.
• Session resumption. If the WebSocket drops, reconnect within 30 seconds and pick up where the conversation left off. Context is preserved.
• Flat-rate pricing. $4.50/hour of session time, no per-token audio surprises, no per-provider invoices, no concurrency commitments. This includes STT, LLM, TTS, turn detection, and tool calling.
• One API to learn. It's one WebSocket—no separate streaming STT connection, LLM HTTP endpoint, TTS stream, and hand-rolled turn detection. The plumbing is in the API, and it works with Claude Code and drop-in LiveKit/Pipecat plugins.
• Built for production. Unlimited concurrency, structured logs per session, and infrastructure where AssemblyAI is considered a business associate under HIPAA and offers a Business Associate Addendum (BAA)—available to sign without a sales call.

Where it fits: AssemblyAI's Voice Agent API is the choice when speech accuracy decides whether the agent ships. It's an API, not a platform—so you keep your architecture and avoid the lock-in of building inside Vapi or Retell. If your agent is taking phone calls, capturing structured data, or operating in a regulated industry where you need a BAA, this is the one to build on.

API vs platform: when to use each

The market splits cleanly into APIs and platforms, and picking the wrong side is the most expensive mistake teams make.

• Choose an API (AssemblyAI Voice Agent API, OpenAI Realtime) when you want to own your architecture, integrate voice into an existing product, avoid platform lock-in, and keep control over your data flow and billing. You call an endpoint from your own stack.
• Choose a platform (Vapi, Retell) when you want a managed environment with prebuilt phone integrations, dashboards, and orchestration handled for you—and you're comfortable building inside someone else's system. Retell in particular is phone-first with strong turn-taking; Vapi is flexible on component choice but adds a platform layer and per-provider accounting.

A useful tell: platforms often let you run a strong STT underneath—Retell, for example, offers Universal-3.5 Pro Realtime in its high-accuracy mode—which means you can get the accuracy without giving up the managed experience. But if you'd rather not build on a platform at all, an API keeps the stack yours.

When to use a chained pipeline instead

A voice agent API is the right answer for most teams in 2026. But there are three cases where chaining the layers yourself still wins:

• You need a specific LLM: A frontier text LLM that isn't exposed inside any voice agent API yet. Most let you choose, but if you need a model that isn't on the list, chain it yourself.
• You need a specific TTS voice: A cloned voice, a specific accent, or a non-standard language model. If you need fine control, a chained pipeline is more flexible.
• You have regulated data residency: Some industries require every layer to run in your VPC. A chained, self-hosted pipeline is the only path.

If you're chaining, the layer that decides whether the agent works is still the streaming STT—the best streaming speech-to-text model for voice agents comes down to the same accuracy and latency criteria above. You can drop AssemblyAI's Streaming Speech-to-Text product into any existing orchestrator.

Common use cases for voice agents

The pattern in 2026 is consistent: voice agents work best on high-volume, structured calls where the agent's job is to capture or look up data rather than reason open-endedly. The teams shipping production agents converge on these use cases:

• Lead qualification and outbound sales discovery: Ask BANT questions, book qualified meetings, sync to the CRM. Turn-taking quality is the differentiator.
• Appointment scheduling and confirmations: Medical offices, salons, service businesses. Alphanumeric accuracy on dates, times, and confirmation codes is non-negotiable.
• Food ordering and reservations: High-accuracy data capture on menu items, special requests, payment info.
• Customer support tier-1 deflection: Order status, account questions, basic troubleshooting—best paired with explicit escalation paths. See our guide to Voice AI for customer service.
• Insurance verification and benefits lookup: Getting plan numbers, group IDs, and member info right the first time—the same accuracy bar that drives voice agents in healthcare.
• Outbound reminders and surveys: Post-visit follow-ups, payment reminders, satisfaction surveys.

The common thread: the agent is capturing or retrieving specific data, the conversation has a predictable structure, and the cost of a transcription error is concrete. That's where a voice agent API earns its keep over a human agent or an IVR.

How to evaluate a voice agent API before you commit

Demos are unreliable. Vendor benchmarks are unreliable. Here's the evaluation loop teams actually use before signing a contract:

1. Record 50 real or representative calls for your use case, including accents, background noise, alphanumeric content, and interruptions.
2. Run them through each API's playground or trial. Measure word error rate on the alphanumeric tokens specifically—phone numbers, confirmation codes, emails, dollar amounts. General WER is misleading. On agent-conversation audio, Universal-3.5 Pro Realtime sits at 6.99% pooled WER and 15.31% entity error rate; run your own audio to see how those numbers hold on your data.
3. Time the turn-taking. Mark every "caller-stops-speaking" moment and measure how long until the agent starts responding. Sub-800ms is the threshold for natural-feeling conversation. Semantic and neural turn detection outperforms basic VAD on hesitant or accented speakers.
4. Test tool calling under load. Define three real tools and have the agent call them mid-conversation. Measure round-trip time and error rate, and note whether the agent speaks naturally during tool execution or goes silent.
5. Read every transcript. You'll catch prompt failures, silently wrong transcriptions, and hallucinated tool parameters that you'd never notice by listening.

Most teams skip step 2 and ship with a model that fumbles confirmation codes silently. Don't.

Final words

The right voice agent API in 2026 depends less on the marketing material and more on what your agent has to actually hear. If your users are on phones, capturing structured data, or operating in regulated environments, the bar is speech accuracy first, latency second, and pricing predictability third—in that order. The chained-architecture APIs (with AssemblyAI's Voice Agent API as the leading example for accuracy-critical use cases) tend to outperform native speech-to-speech models on real-world telephony, even when the native models look better in studio-audio demos.

The subtler decision is API versus platform. Universal-3.5 Pro Realtime's accuracy edge is now available both ways—directly through the AssemblyAI Voice Agent API, and inside platforms like Retell that run it underneath—so you can pick the buying model that fits your team without giving up the accuracy that decides whether the agent ships. For most teams shipping this year, the AssemblyAI Voice Agent API is the right starting point: one WebSocket, $4.50/hour, Universal-3.5 Pro Realtime for the parts that matter, and no platform lock-in.

Frequently asked questions

What is the best voice agent API in 2026?

The best voice agent API in 2026 is AssemblyAI's Voice Agent API for production deployments where speech accuracy matters—a single WebSocket built on Universal-3.5 Pro Realtime with 6.99% pooled word error rate on real agent conversations, ~300ms turn detection, context carryover, tool calling with intermediate speech, and flat $4.50/hour pricing with no concurrency commitments. In our Voice Agent Report, 76% of builders rated transcription accuracy as the most important non-negotiable, and Universal-3.5 Pro Realtime posts the lowest entity error rate (15.31%) against Deepgram Flux (50.50%) and ElevenLabs Scribe v2 (39.70%). OpenAI Realtime is competitive for browser-first demos, and Retell is competitive for phone-first agents that prize turn-taking. The right choice depends on whether your users are on phones, what data the agent captures, and how predictable you need pricing to be.

Which voice agent API has the best speech accuracy?

On the Pipecat open STT benchmark—built from real agent conversations rather than studio clips—AssemblyAI's Universal-3.5 Pro Realtime (the model under its Voice Agent API) posts the best speech accuracy: a 6.99% pooled word error rate versus 9.04% for Google Chirp3, 9.76% for ElevenLabs Scribe v2, and 15.58% for Deepgram Flux. Its entity error rate is 15.31%, roughly a third of Deepgram Flux's 50.50%, and it captures phone numbers at 3.55% error. For voice agents, entity accuracy on phone numbers, confirmation codes, and names is the number that predicts whether the agent ships—general WER hides it.

What is the best voice agent API for developers?

For developers, AssemblyAI's Voice Agent API is the strongest fit: it's one WebSocket (no juggling separate STT, LLM, and TTS connections), needs no SDK, works with Claude Code, and ships drop-in plugins for LiveKit and Pipecat. You can update the system prompt, voice, tools, and VAD mid-session with a single JSON message, and resume a dropped session within 30 seconds without losing context. By contrast, OpenAI Realtime spreads work across 30+ event types, and platforms like Vapi and Retell ask you to build inside their environment. If you want a clean API you call from your own stack, an API beats a platform.

How much do voice agent APIs cost?

Voice agent API pricing falls into a few models. AssemblyAI's Voice Agent API is flat $4.50/hour, all-in—STT, LLM, TTS, turn detection, and tool calling—with no concurrency commitments. OpenAI Realtime bills per token across 30+ event types and runs roughly $18/hour, about 4x more and hard to forecast at scale. Platforms like Retell and ElevenLabs Conversational AI charge per minute, roughly $0.07–0.30/min (about $4.20–$18/hour), and ElevenLabs prices at the higher end and caps concurrency around 30. Vapi charges a platform fee plus pass-through provider costs. Flat hourly pricing is the easiest to model: one line of math tells you what a 5-minute call costs.

API vs platform (Vapi/Retell)—which should I choose?

Choose an API (like the AssemblyAI Voice Agent API or OpenAI Realtime) when you want to own your architecture, integrate voice into an existing product, and avoid platform lock-in—you call an endpoint from your own stack. Choose a platform (Vapi or Retell) when you want a managed environment with prebuilt phone integrations, dashboards, and orchestration handled for you, and you're fine building inside their system. Retell is phone-first with strong turn-taking; Vapi is flexible on components but adds a platform layer and per-provider accounting. Note that platforms can run a strong STT underneath—Retell offers Universal-3.5 Pro Realtime in its high-accuracy mode—so you don't always trade accuracy for convenience.

What is a speech-to-speech voice agent?

A speech-to-speech voice agent is a voice agent exposed behind a single API—usually a WebSocket—that takes a user's audio in and returns the agent's audio out, hiding transcription, reasoning, tool calls, and voice synthesis behind one connection. There are two designs: a chained (cascading) speech-to-speech pipeline that runs streaming STT into an LLM into TTS but exposes one API (AssemblyAI's Voice Agent API), and a native speech-to-speech model trained end-to-end on audio (OpenAI Realtime). Chained pipelines give you more transparency and let you swap best-in-class layers; native models pitch lower latency and richer audio understanding but offer less control and narrower language coverage.

Do voice agent APIs support multiple languages?

Yes, but coverage varies widely. AssemblyAI's Voice Agent API runs on Universal-3.5 Pro Realtime, which covers 18 languages—English, Spanish, French, German, Italian, Portuguese, Arabic, Danish, Dutch, Hebrew, Hindi, Japanese, Mandarin, Vietnamese, Finnish, Norwegian, Swedish, and Turkish—with mid-sentence code-switching. OpenAI Realtime supports around 50 languages but has hallucination and language-switching issues mid-call. If you need a specific language combination, test with real audio in those languages before you commit—support varies significantly between studio benchmarks and real-world phone audio.

‍