AssemblyAI vs Deepgram: what's the best voice agent API?

AssemblyAI and Deepgram both let you build production voice agents, but they take very different bets on the hardest part—hearing the caller correctly and knowing when they're done talking. AssemblyAI ships a full STT+LLM+TTS pipeline behind one WebSocket at a flat $4.50/hr, built on Universal-3.5 Pro Realtime. Deepgram splits its story between Nova-3 and its newer Flux "conversational speech recognition" model, with tiered pricing and concurrency charges. This is an honest, voice-agent-specific comparison.

If you want the broader transcription-first view, we cover that in our general AssemblyAI vs Deepgram comparison. This post stays squarely on building voice agents: realtime accuracy on the entities that make calls succeed, turn detection and latency, and how each vendor prices and packages the pipeline.

The short version

For voice agents, the gap is not subtle. On the Pipecat open STT benchmark—real agent conversations, not read-aloud audiobooks—Universal-3.5 Pro Realtime posts a pooled word error rate of 6.99% versus 15.58% for Deepgram Flux. On entities, the numbers that decide whether an agent books the right appointment or charges the right card, the spread is larger: 15.31% entity error rate for AssemblyAI versus 50.50% for Flux. Names: 16.92% vs 39.21%. Phone numbers: 3.55% vs 10.41%. If your agent takes orders, verifies identities, or reads back numbers, that difference is the whole ballgame.

Voice agent comparison at a glance

Two philosophies: conversational speech recognition vs a full agent pipeline

Deepgram's pitch for agents centers on Flux, which it markets as "conversational speech recognition"—a model that leans heavily on turn detection to decide when the caller has finished speaking. It's a reasonable idea, but it downplays raw transcription accuracy (Deepgram's public accuracy claims stay vague, in the ">90%" range), and it's still fundamentally a speech-recognition component you wire into your own agent stack.

AssemblyAI takes the pipeline all the way. The Voice Agent API is a complete STT+LLM+TTS loop behind a single WebSocket, built on Universal-3.5 Pro Realtime as the speech foundation. You don't stitch together three vendors, reconcile three invoices, or debug three failure surfaces. One connection, a handful of JSON messages, roughly one second end-to-end, and it works natively with tools like Claude Code—no SDK required. Most developers ship a working agent the same afternoon.

Both approaches use dedicated speech models rather than a single multimodal model, which is the right call for production agents—the transcript feeds the LLM that decides what to do next, so speech quality is upstream of everything. The difference is how much of that pipeline you have to assemble yourself, and how accurate the speech layer is once you do.

Speech accuracy: where the gap shows up

Voice agents don't transcribe for the record—they transcribe to act. If the speech layer drops a digit in a phone number, the agent texts the wrong person. If it garbles a name during identity verification, the call stalls. So the benchmark that matters is realtime accuracy on real conversations, especially on entities.

On the Pipecat open STT benchmark of real agent conversations, Universal-3.5 Pro Realtime leads the market with a 6.99% pooled WER. Against Deepgram Flux, the entity story is where it becomes decisive:

This isn't a theoretical edge. HotelPlanner switched to AssemblyAI from Deepgram specifically over credit-card and number accuracy on live calls—accurate readback of payment details is the difference between a completed booking and an abandoned one. After switching, they saw a 10% lift in checkout conversion. When the agent hears the numbers right the first time, callers don't hang up.

Accuracy also compounds when you feed the model context. With agent_context—passing the agent's own question so the model interprets the reply through that lens—AssemblyAI cut WER by 10.2% across 20,000 voice-agent files, with short-utterance errors down 13.7% and name-entity errors down 9.4%. One team combined agent context with contextual prompting and took their production utterance error rate from 26% to 9%. Rolling conversation memory is on by default, so the model already remembers what was said earlier in the call with nothing to configure. Deepgram has no equivalent to this context layer.

Turn detection and latency: the feel of the conversation

Turn detection is where voice agents live or die, and it's the exact problem Deepgram built Flux around. AssemblyAI's approach reads tonality, pacing, and rhythm to distinguish a thoughtful pause from a finished thought, landing an end-of-turn decision around 300ms. It's baked into Universal-3.5 Pro Realtime, so it draws on the same acoustic understanding that drives transcription accuracy, and it supports natural barge-in—cut in, and the agent stops and listens.

Latency is the other half of "feel," and it's measurable. On the Pipecat benchmark, AssemblyAI posts a ~262ms mean time-to-transcript with a p95 of 355ms; Deepgram comes in at 568ms mean with a p95 of 1319ms. That p95 gap matters more than the mean—it's the tail latency callers experience as awkward silence before the agent responds. Lower and more consistent latency keeps conversations natural instead of stilted.

The honest recommendation stands: build a quick agent on each and have a real conversation. Turn feel is something you notice in the first thirty seconds, even when it's hard to put in a table.

Pricing and packaging

AssemblyAI's Voice Agent API is a flat $4.50/hr for the full pipeline—STT, LLM reasoning, and voice generation. No separate token charges, no per-feature add-ons on the agent price, no commit, and unlimited concurrency with no rate limits. Your cost model is hours times $4.50, full stop, which makes budgeting for bursty traffic trivial.

Deepgram uses tiered, usage-based pricing and charges extra for concurrency. For a support line that spikes at peak hours, concurrency metering makes costs harder to predict and potentially more expensive exactly when you're busiest. You're also assembling and paying for the agent pipeline around a speech component rather than buying the whole loop at one rate. If you'd rather not model concurrency curves, flat pricing is one less thing to forecast. For current numbers on either side, check the AssemblyAI pricing page.

Developer experience

Both APIs use WebSocket connections and JSON messages, so the fundamentals rhyme. But AssemblyAI's Voice Agent API is deliberately minimalist: a handful of message types, no SDK required, and the whole reference readable in about ten minutes. Because it's designed to work with tools like Claude Code, you can paste the docs in and scaffold a working integration quickly. There are drop-in plugins for LiveKit and Pipecat, so you can slot it into an existing stack.

AssemblyAI also supports live mid-conversation updates—change the system prompt, swap voices, add or remove tools, and adjust turn settings via a JSON message without dropping the connection. For a support agent escalating from English to Spanish or a coaching app switching modes mid-call, that's a real advantage. Deepgram's DX is solid and conventional; if you're already deep in Deepgram's transcription products, adding their agent tooling is a natural extension.

Which should you choose?

If your agent handles anything that has to be right the first time—names, numbers, addresses, payment details—AssemblyAI's Voice Agent API is the stronger foundation, and the realtime benchmarks aren't close. The 6.99% vs 15.58% WER gap and the 15.31% vs 50.50% entity gap translate directly into calls that complete without a "sorry, can you repeat that?"

Deepgram Flux is a credible bet if you're standardizing on Deepgram and turn detection is your primary concern. But notice the pattern: the thing Flux optimizes for—knowing when the caller stopped—is only useful if you also heard what they said. AssemblyAI wins on both, ships the full pipeline, and charges one flat rate. For teams starting fresh on voice agents that work reliably in production, that combination is hard to beat.

Frequently asked questions

AssemblyAI vs Deepgram for voice agents—which is better?

AssemblyAI is the stronger choice for most production voice agents. On the Pipecat open STT benchmark of real agent conversations, Universal-3.5 Pro Realtime posts a 6.99% word error rate versus 15.58% for Deepgram Flux, and a 15.31% entity error rate versus 50.50%. AssemblyAI also ships a full STT+LLM+TTS pipeline behind one WebSocket at a flat $4.50/hr, while Deepgram provides a speech component you assemble into your own stack with tiered, concurrency-metered pricing.

Deepgram Flux vs Universal-3.5 Pro Realtime—how do they compare?

Universal-3.5 Pro Realtime leads Deepgram Flux across the board on agent conversations: 6.99% vs 15.58% word error rate, 15.31% vs 50.50% entity error rate, 16.92% vs 39.21% on names, and 3.55% vs 10.41% on phone numbers. Deepgram markets Flux as "conversational speech recognition" focused on turn detection, but it downplays raw accuracy. AssemblyAI matches the turn-detection story—end-of-turn decisions land around 300ms—while also winning decisively on transcription quality.

Which is more accurate on names and numbers in calls?

AssemblyAI. On agent conversations, Universal-3.5 Pro Realtime hits a 16.92% name error rate versus Deepgram Flux's 39.21%, and a 3.55% phone-number error rate versus 10.41%. This is why HotelPlanner switched from Deepgram over credit-card accuracy and saw a 10% lift in checkout conversion—when the agent reads numbers back correctly the first time, callers complete the task instead of hanging up.

Which has lower latency for voice agents?

AssemblyAI. On the Pipecat benchmark, AssemblyAI posts a ~262ms mean time-to-transcript with a p95 of 355ms, versus Deepgram's 568ms mean and 1319ms p95. The p95 gap matters most—it's the tail latency callers experience as awkward silence. End-of-turn detection lands around 300ms, keeping conversations natural rather than stilted.

How does AssemblyAI vs Deepgram voice agent pricing compare?

AssemblyAI charges a flat $4.50/hr for the full STT+LLM+TTS pipeline with no commit, no per-feature add-ons on the agent price, and unlimited concurrency. Deepgram uses tiered, usage-based pricing and charges extra for concurrency, which makes costs harder to predict during traffic spikes. For teams with bursty support traffic, AssemblyAI's flat model is simpler to budget. Check the pricing page for current numbers on either side.

Can I build a full voice agent on each?

With AssemblyAI, yes—out of the box. The Voice Agent API is a complete STT+LLM+TTS loop behind one WebSocket, roughly one second end-to-end, built on Universal-3.5 Pro Realtime, with drop-in LiveKit and Pipecat plugins and no SDK required. Deepgram provides the speech recognition component (Flux or Nova-3); you assemble the LLM and TTS around it, manage the integrations, and handle concurrency in your billing model.

‍