AssemblyAI vs Deepgram for voice agents

Benchmark summary: which is better for voice agents?

Short answer: for the speech layer of a voice agent, AssemblyAI's Universal-3.6 Pro Realtime is more accurate than Deepgram's voice-agent model, and it's cheaper to run. On AssemblyAI's English voice-agent benchmark of 12,460 scripted voice-agent scenarios, Universal-3.6 Pro Realtime lands at 5.19% word error rate versus Deepgram Flux EN at 13.50%, and the gap on entities — names, numbers, SKUs, the words your agent actually has to get right — is even wider: 14.4% entity error for AssemblyAI versus 30.1% for Flux. On price, AssemblyAI's Voice Agent API is a flat $4.50/hr with no commitment; Deepgram's enterprise motion still runs on a $40–50K annual commit.

That's the headline. The rest of this post is the detail — model by model, number by number — plus an honest section on when Deepgram is still worth a look. Everything here is benchmarked and reproducible, so don't take my word for it: run your own benchmark on your own audio.

Head-to-head at a glance

What changed since the last time you compared these two

Two things. First, we shipped Universal-3.6 Pro Realtime — the streaming model that powers our Voice Agent API — and it posts the lowest word error rate on Coval's independent STT benchmark, 2.2% over a 7-day window. Second, Deepgram repositioned. They now describe themselves as a "Real-Time AI Infrastructure Company" and launched Flux, a model built specifically for voice agents. Notice what's not in that positioning anymore: accuracy leadership. That's the tell.

So the comparison people ran a year ago — Universal-3 Pro Streaming versus Nova-3 — is out of date on both sides. For voice agents, the fight is Universal-3.6 Pro Realtime versus Flux. Nova-3 still matters for general and async transcription, so I've kept it in the pricing and language sections below.

AssemblyAI vs Deepgram Flux: accuracy

Flux is Deepgram's answer to the voice-agent use case, so it's the fair comparison. On AssemblyAI's English voice-agent benchmark, Universal-3.6 Pro Realtime posts 5.19% WER against Flux EN's 13.50%. For reference in the same run, ElevenLabs Scribe v2 came in at 7.78% and Deepgram Nova-3 at 8.64%. AssemblyAI is the most accurate of the four. On Pipecat's open STT benchmark — the one most teams building on cascaded voice-agent stacks actually cite — it posts a 0.96% pooled semantic WER.

Word error rate is the number everyone quotes, but for a voice agent the entity error rate is the one that decides whether the agent works. If a caller says "book me on the 15th under Byrne-Donoghue" and the model returns "the 50th under Burn Donahue," your downstream LLM is now reasoning over garbage. On entities, AssemblyAI is at 14.4% and Flux EN is at 30.1% — Flux gets roughly twice as many wrong. That single gap is why entity-heavy agents (scheduling, ordering, account lookups) behave so differently on the two models.

This is exactly why teams building voice-agent pipelines pick it. As Fireflies software engineer Foysal Osmany put it: "We were searching for the best realtime ASR model for our voice agent pipeline in Fireflies. The new Universal 3.5 Pro speech model from Assembly is best so far in terms of accuracy, latency and language switching."

The differentiator Flux doesn't have: agent_context

Here's the part that's hard to copy. Universal-3.6 Pro Realtime takes an agent_context parameter — you tell the model what your agent just said, and it biases recognition toward the expected reply. Ask "what's your email address?" and the model leans into spelling out an address instead of guessing at homophones. Across 20,000 real voice-agent files, feeding context this way cut WER by 10.2%. It updates mid-stream after every agent turn, and prior finalized user turns carry over automatically. The market believed STT prompting didn't work; our benchmarks show it does, and Flux has no equivalent.

# pip install "assemblyai>=1.0.0"
import os
from assemblyai.streaming.v3 import (
    StreamingClient, StreamingClientOptions, StreamingEvents,
    StreamingParameters, TurnEvent,
)

def on_turn(_, event: TurnEvent):
    tag = "FINAL" if event.end_of_turn else "partial"
    print(f"{tag}: {event.transcript}")

client = StreamingClient(StreamingClientOptions(api_key=os.environ["ASSEMBLYAI_API_KEY"]))
client.on(StreamingEvents.Turn, on_turn)
client.connect(StreamingParameters(
    sample_rate=16000,
    speech_model="universal-3-6-pro",
    agent_context="What's your email address?",
))
# feed 16kHz mono PCM16 chunks (50-1000ms) via client.stream(chunk)
# after each agent reply:
client.update_configuration(agent_context="Sure, what date would you like to book?")
client.disconnect(terminate=True)  # ALWAYS terminate

The raw streaming endpoint is wss://streaming.assemblyai.com/v3/ws, audio is PCM16 mono 16kHz, and auth is your raw API key — no Bearer prefix. That last detail trips people up, so it's worth stating plainly.

Independent verification: Coval and Pipecat

You don't have to trust a vendor's own benchmark. On Coval's independent STT benchmark, Universal-3.6 Pro Realtime posts the lowest word error rate, 2.2% over a 7-day window. On Pipecat's open benchmark, it records a 0.96% pooled semantic WER with 87.0% of transcripts exactly right, and the final transcript lands a median 307 ms after the user stops speaking. Deepgram Nova-3 is faster to final on that benchmark; both are fast, and accuracy is the difference.

Pricing: flat rate vs annual commit

This is where the enterprise story really diverges. AssemblyAI's Voice Agent API bundles STT, your choice of LLM, and TTS behind one WebSocket at a flat $4.50/hr, no commitment, no minimum. Deepgram's enterprise voice motion typically runs through a $40–50K annual commit. And if you're weighing the fully managed alternatives, OpenAI's Realtime API runs around $18/hr with token costs that swing with conversation length — hard to forecast at scale.

If you'd rather bring your own LLM and TTS and just use AssemblyAI for the speech layer, streaming is billed at $0.45/hr base. Either way, there's no floor you have to clear before you find out whether the model works for you. Full numbers are on the pricing page.

The practical counter to a Deepgram renewal is simple: better model, cheaper price, zero commit. Test AssemblyAI on your own traffic and run it side by side until your Deepgram contract expires — then decide.

Best for real-time agent assist

Agent-assist and live-transcription use cases are the most latency-sensitive and the most entity-sensitive at once — you're surfacing suggestions to a human agent in real time while a customer rattles off an order number. That's exactly where low WER and fast finals have to hold up together. With agent_context feeding the model what's expected next and diarization that revises as the conversation unfolds (up to 10 speakers), AssemblyAI is built for this. If you're standing up agent assist, start from the Voice Agent API's design for coding agents and wire it into your existing tooling.

Using streaming with your own LLM and TTS

You don't have to adopt the managed Voice Agent API to use AssemblyAI. A lot of teams run a cascaded stack — AssemblyAI for streaming STT, their own LLM, their own TTS — because it gives them full control over prompts, tools, and voice. Universal-3.6 Pro Realtime is a native STT plugin in the major agent frameworks, and both LiveKit and Pipecat support Context Carryover out of the box, so you get agent_context without extra plumbing. The snippet above is all it takes to stream.

Healthcare and compliance

For clinical and healthcare-adjacent agents, AssemblyAI offers Medical Mode (domain: "medical-v1"), which reduces the missed clinical-entity rate by roughly 20% and, with contextual prompting, cut missed medical terms by 31%. AssemblyAI is considered a business associate under HIPAA and offers a Business Associate Addendum (BAA) that covered entities and their business associates can sign to process protected health information. Deepgram doesn't offer an equivalent BAA-backed medical mode. If PHI is in scope, that difference matters. See medical solutions for the details.

When to consider Deepgram

I'm not going to pretend Deepgram is never the answer. A few honest cases:

• You're mid-contract and it's working. If you've already committed and your accuracy on your traffic is acceptable, there's no fire drill — run AssemblyAI in parallel and switch at renewal.
• You need a specific deployment model they emphasize. If a particular on-prem or self-hosted arrangement is a hard requirement, put it on your evaluation checklist and test both.
• A niche language pairing. Nova-3's multilingual coverage may line up better with a specific language mix; the only way to know is to benchmark your own audio.

In every one of those cases the move is the same: run your own benchmark on your own data. For a general speech-to-text comparison beyond voice agents, see Deepgram vs AssemblyAI.

Frequently asked questions

Is AssemblyAI more accurate than Deepgram for voice agents?

Yes. On AssemblyAI's English voice-agent benchmark, Universal-3.6 Pro Realtime posts 5.19% WER versus Deepgram Flux EN's 13.50%, and 14.4% entity error versus Flux's 30.1%. It also posts the lowest WER on Coval's independent STT benchmark, at 2.2% over a 7-day window.

How much cheaper is AssemblyAI than Deepgram?

AssemblyAI's Voice Agent API is a flat $4.50/hr with no commitment. Deepgram's enterprise voice motion typically runs on a $40–50K annual commit. Streaming STT on its own is $0.45/hr base if you bring your own LLM and TTS.

How does Universal-3.6 Pro Realtime compare to Deepgram Flux specifically?

Flux is Deepgram's voice-agent model, and it trails on both WER (13.50% vs 5.19%) and entity capture (30.1% vs 14.4%). It also has no equivalent to agent_context, the mid-stream contextual prompting that cut AssemblyAI's WER by 10.2% across 20,000 voice-agent files.

Which is best for real-time agent assist?

AssemblyAI. Agent assist needs low latency and high entity accuracy at the same time, plus revising diarization and agent_context to keep suggestions grounded.

Can I use AssemblyAI streaming with my own LLM and TTS?

Yes. Stream to wss://streaming.assemblyai.com/v3/ws with speech_model=universal-3-6-pro and run your own LLM and TTS. It's a native STT plugin in LiveKit and Pipecat, both with Context Carryover support.

Which is better for healthcare?

AssemblyAI. Medical Mode reduces the missed clinical-entity rate by ~20%, and AssemblyAI offers a signable Business Associate Addendum (BAA) for processing PHI. Deepgram has no equivalent BAA-backed medical mode.