What is speech recognition? A 2026 guide to STT APIs

Speech recognition is the technology that turns spoken audio into text a computer can use. If you're reading this, you probably don't just want the definition — you want to know how to pick a speech-to-text API, wire it into an app, and trust the numbers. This guide covers both: a tight explanation of how speech recognition works, then the decision and integration content most buyer's guides skip.

What is speech recognition?

Speech recognition — also called automatic speech recognition (ASR) or speech-to-text (STT) — converts an audio signal of human speech into written words. A modern system samples the audio, extracts acoustic features, and uses a trained neural model to predict the most likely sequence of words, then adds punctuation, casing, and formatting so the output reads like text a person wrote. For the deeper mechanics, see what is ASR and the speech-to-text cornerstone.

A quick terminology note people ask about: speech recognition converts speech to text (what did they say?), while voice recognition usually means identifying who is speaking. They overlap — speaker diarization, below, lives in that overlap — but they answer different questions.

Modern speech recognition rarely stops at a raw transcript. On top of the words, Speech Understanding models add speaker labels, PII redaction, and structured analysis so you get meaning, not just a wall of text.

How do I choose the best speech-to-text API?

There's no single "best" — there's best for your audio, your latency budget, and your compliance needs. The honest answer: benchmark the top candidates on your own recordings. Vendor accuracy numbers are measured on clean data that rarely resembles a drive-through, a noisy call center, or a three-person meeting. Here's the checklist we'd use, in priority order:

• Accuracy on your audio. Run a representative sample through each API and compute word error rate (WER) yourself. Our guide on evaluating speech recognition models shows the method. For reference, Universal-3.5 Pro posts a 7.69 normalized WER on code-switching audio, and Universal-3.5 Pro Realtime hits 6.99% WER on the Pipecat voice-agent benchmark — but you should still confirm on your data.
• Latency tier. Batch, sync, or streaming — pick the one that matches your product (see the latency section below).
• Speaker diarization. If you need to know who said what, compare diarization accuracy specifically (details below).
• Language coverage and code-switching. Universal-3.5 Pro handles native code-switching across 18 languages and falls back to Universal-2 for 99+ total.
• Robustness to bad audio. Test on your worst network streams and noisiest rooms, not just clean samples.
• Speech Understanding features. PII redaction, speaker labels, and downstream analysis you'd otherwise build yourself.
• Security and compliance. Covered in its own section below.
• Price and predictability. Universal-3.5 Pro is $0.21/hr async; Universal-3.5 Pro Realtime is $0.45/hr base; legacy Universal-2 is $0.15/hr. See pricing and the benchmarks to compare.

Public teams run this exact process. CallRail, the call-tracking and marketing-analytics platform, builds on AssemblyAI — as Chief Product Officer Ryan Johnson puts it: "The capabilities AssemblyAI enables us to build help businesses market confidently while saving time and money. It's powerful, almost magical to see it work." Conversation-intelligence platform Jiminny tells a similar story; founder and CEO Tom Lavery: "AssemblyAI has a real high-touch personal service. It's a great partnership and we're very collaborative and get to test new AI models early and work together. And AssemblyAI is really pushing boundaries, helping us create a well-rounded conversation intelligence platform." The pattern holds — the teams that benchmark on their own audio end up happiest.

How do I invoke a speech-to-text API from a mobile app?

Two integration paths, and the right one depends on whether you're processing a recording or a live mic:

• Pre-recorded files → REST. Upload or point the API at an audio URL over HTTPS and poll (or use a webhook) for the result. The REST base is https://api.assemblyai.com (or https://api.eu.assemblyai.com for EU data residency).
• Live audio → WebSocket. Stream mic audio over a persistent connection and receive transcripts turn by turn. The streaming endpoint is wss://streaming.assemblyai.com/v3/ws.

The critical mobile rule: never ship your API key in the app. Anyone can extract it from a binary. Instead, proxy file uploads through your own server, and for streaming, mint a short-lived token server-side and pass it to the client:

# Server-side: mint a temporary streaming token
GET https://streaming.assemblyai.com/v3/token?expires_in_seconds=60

# Client connects with the token instead of your API key:
wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&speech_model=universal-3-5-pro&token=<TEMP_TOKEN>

Here's the file path in Python — this is the same call whether it runs on your backend for an iOS or Android client. Note the raw API key with no Bearer prefix, and the plural speech_models fallback list:

import assemblyai as aai, os

aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],  # optional; this is the default
    speaker_labels=True,
)
transcript = aai.Transcriber(config=config).transcribe("https://assembly.ai/wildfires.mp3")
if transcript.status == aai.TranscriptStatus.error:
    raise RuntimeError(transcript.error)
print(transcript.text)

And the live path in Python (streaming SDK v3) — singular speech_model, and always terminate the session when you're done:

import os
from assemblyai.streaming.v3 import (
    StreamingClient, StreamingClientOptions, StreamingEvents, StreamingParameters,
    TurnEvent,
)

def on_turn(_, event: TurnEvent):
    tag = "FINAL" if event.end_of_turn else "partial"
    print(f"{tag}: {event.transcript}")

client = StreamingClient(StreamingClientOptions(api_key=os.environ["ASSEMBLYAI_API_KEY"]))
client.on(StreamingEvents.Turn, on_turn)
client.connect(StreamingParameters(sample_rate=16000, speech_model="universal-3-5-pro"))
# feed 16kHz mono PCM16 chunks (50-1000ms) via client.stream(chunk)
client.disconnect(terminate=True)  # ALWAYS terminate

For the complete parameter list and language SDKs, see the API reference.

Speech recognition latency: batch vs sync vs streaming

Latency is the axis that usually decides the API, so match the tier to the product:

• Async (batch): best for files where a few seconds of processing is fine — meeting recordings, podcasts, call archives. Highest accuracy, no latency pressure.
• Sync API (~100ms): a middle tier for request/response transcription where you want a fast answer on a short clip without holding a streaming connection open.
• Streaming (~110ms): for live conversations — voice agents, live captioning, agent assist. Streaming speech-to-text on Universal-3.5 Pro Realtime lands in Coval's independent "Human Parity Zone" at 3.40% WER with ~110ms p50 time-to-final-segment.

Rule of thumb: if the user is waiting to speak again, you need streaming. If a machine is waiting on a short clip, sync fits. Everything else is async.

Speaker diarization: how it works and how to compare it

Speaker diarization answers "who spoke when" by segmenting audio and clustering segments by speaker, so your transcript reads as a labeled conversation rather than an undifferentiated block. It's essential for meeting notes, call analytics, and any multi-party audio.

The number to compare is concatenated minimum-permutation WER (cpWER), which folds speaker-attribution errors into the accuracy score — lower is better. Universal-3.5 Pro is the most accurate diarization we've shipped, at an average cpWER of 30.17, versus 37.92 for Deepgram Nova-3, 35.26 for ElevenLabs Scribe v2, and 36.87 for Gladia. Turn it on with speaker_labels=True. When you compare vendors, run cpWER on your own multi-speaker audio — single-speaker WER hides diarization errors entirely.

How AssemblyAI compares to Google, AWS, and Azure

The big-cloud STT services are convenient if you're already in their ecosystem, but they trail on accuracy, prompting, and predictable pricing. AssemblyAI ships contextual prompting (most competitors don't), a promptable streaming model, and diarization tuned for cpWER — with flat, published per-hour pricing rather than opaque tiers. The right way to settle it is a head-to-head on your audio; the benchmarks page is the starting point, and our guide on how accurate speech-to-text is shows how to read the results.

Security and compliance

For regulated workloads, check data residency, encryption, and healthcare terms before you build. AssemblyAI offers EU data residency and, for healthcare, a Business Associate Addendum: AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a Business Associate Addendum (BAA) that can be signed in minutes. For clinical audio specifically, Medical Mode (domain="medical-v1") reduces missed clinical-entity rates.

Get started

Get your free API key and transcribe your first file in minutes. Exploring beyond transcription? Explore Voice AI solutions, or compare the numbers on the benchmarks page.

Frequently asked questions

How do I choose the best speech-to-text API?

Benchmark the top candidates on your own audio and compute WER yourself, then weigh latency tier, diarization accuracy (Universal-3.5 Pro averages 30.17 cpWER), language coverage, and compliance. Vendor numbers are measured on clean data — your recordings are the real test.

How do I invoke a speech-to-text API from a mobile app?

Use REST for pre-recorded files and a WebSocket for live audio. Never embed your API key in the app — proxy file uploads through your server and mint a short-lived streaming token server-side for live capture.

What's the difference between speech recognition and voice recognition?

Speech recognition converts speech to text (what was said); voice recognition typically identifies who is speaking. Speaker diarization sits in the overlap, labeling who said what within a transcript.

How accurate is speech recognition today?

The best models are near human parity. Universal-3.5 Pro Realtime sits in Coval's independent Human Parity Zone at 3.40% WER, and Universal-3.5 Pro posts 7.69 normalized WER on hard code-switching audio.

Can speech recognition run in real time?

Yes. Streaming delivers transcripts at ~110ms p50 time-to-final-segment, and a Sync API tier (~100ms) covers short request/response clips. Async is best when latency isn't a constraint.

Which speech-to-text API has the best diarization?

Compare on cpWER. Universal-3.5 Pro averages 30.17 cpWER versus 37.92 (Deepgram Nova-3), 35.26 (ElevenLabs Scribe v2), and 36.87 (Gladia) — verify on your own multi-speaker audio.

‍

‍