Python Speech Recognition in 2026

‍Python speech recognition in 2026 comes down to two paths: open-source libraries like Whisper, faster-whisper, and Vosk for local or offline work, and cloud APIs like AssemblyAI for the highest real-world accuracy with the least infrastructure. This guide covers both — with runnable code — so you can pick the right tool and ship.

The speech recognition market is projected to reach $53.67 billion by 2030, and Python sits at the center of nearly every build. But the "right" library depends entirely on your constraints: offline or cloud, real-time or batch, tight budget or top accuracy. Let's walk through all of it.

What is speech recognition?

Speech recognition converts spoken language into text using AI models. The metric that matters is word error rate (WER) — the percentage of words transcribed incorrectly. Production systems typically land below 10% WER on clean audio, but real-world performance with noise and diverse accents can run two to three times higher. If you're benchmarking, it's worth understanding why WER alone can mislead you.

Open-source vs. cloud-based Python speech recognition

Open-source Python speech recognition options

OpenAI Whisper

Whisper spans 39M parameters (tiny) to 1.5B (large-v3), plus a turbo variant, and handles 99 languages, noisy audio, and multiple speakers. The drawbacks: it runs slower than real-time on CPU, and it has a documented hallucination tendency — Cornell researchers found roughly 1% of transcriptions contain fabricated phrases. AssemblyAI's Universal models show around a 30% reduction in hallucination rates versus Whisper large-v3, and Whisper tends to lag on proper-noun detection.

Faster-whisper and Distil-Whisper

Faster-whisper uses the CTranslate2 inference engine for up to 4x faster transcription at comparable accuracy and lower memory use, supporting the same models and languages as Whisper. Distil-Whisper (Hugging Face) runs about 6x faster than Whisper while retaining ~99% of its accuracy, and is especially effective for English.

Vosk

Vosk ships lightweight, language-specific models (50MB to 1.8GB) across 20+ languages, with real-time recognition and live microphone input that runs on a Raspberry Pi. The trade-off is lower accuracy than Whisper or cloud APIs, and it struggles with overlapping speakers and heavy background noise.

SpeechRecognition

The SpeechRecognition library is a unified Python wrapper over backends including Google Cloud Speech-to-Text, CMU Sphinx, Wit.ai, Azure, Houndify, IBM Watson, Vosk, and Whisper. It's handy for prototyping, but it adds an abstraction layer without adding capability — you get limited exposure to each engine's advanced options.

wav2letter and DeepSpeech (legacy)

wav2letter (now Flashlight, from Facebook AI Research) uses a CNN-based architecture but requires manual C++ compilation and sees minimal community activity. DeepSpeech (Mozilla) is archived and no longer maintained — the ecosystem has moved to Whisper. Mention them for completeness, not for new builds.

Cloud Python speech recognition with AssemblyAI

When you need the highest accuracy with the least infrastructure, a cloud API wins. AssemblyAI's flagship async model is Universal-3.5 Pro (universal-3-5-pro), which delivers native code-switching across 18 languages and a 7.69% average code-switching word error rate — ahead of ElevenLabs Scribe v2 (8.77), the previous-generation Universal-3 Pro (9.07), and Deepgram Nova-3 (12.22) on the same benchmark. Its diarization is the most accurate AssemblyAI has shipped, at 30.17 average cpWER. Full numbers live on the benchmarks page.

Using AssemblyAI's speech-to-text API

Install the SDK with pip install assemblyai, then transcribe a file. Note the speech_models list — it's an ordered fallback, so you get the latest flagship first and a stable model if anything is unavailable:

# pip install assemblyai
import assemblyai as aai
import os

aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],  # latest first, stable fallback
    speaker_labels=True,
)
transcript = aai.Transcriber(config=config).transcribe("https://assembly.ai/wildfires.mp3")

if transcript.status == aai.TranscriptStatus.error:
    raise RuntimeError(transcript.error)
print(transcript.text)

That's a production-ready transcript in a dozen lines, with speaker diarization already enabled.

Ready to build? Get a free API key and run your first file — no credit card required. Prefer to explore first? Open the playground.

Speaker diarization with AssemblyAI

Diarization identifies who spoke when in multi-speaker audio — essential for meetings, podcasts, and call centers. It's the speaker_labels=True flag shown above (a +$0.02/hr async add-on). Once the transcript completes, you can iterate over labeled utterances:

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

Multilingual speech recognition

Universal-3.5 Pro handles 99+ languages (via the Universal-2 fallback tier for the long tail) and native code-switching. To let the model detect the language automatically, set language_detection=True:

config = aai.TranscriptionConfig(
    speech_models=["universal-3-5-pro", "universal-2"],
    language_detection=True,
)
transcript = aai.Transcriber(config=config).transcribe("https://assembly.ai/wildfires.mp3")
print(transcript.text)

‍

Getting paragraphs and sentences

The API automatically segments transcripts with timestamps and speaker labels. Use get_sentences() and get_paragraphs():

for paragraph in transcript.get_paragraphs():
    print(paragraph.text)

Real-time speech recognition in Python

For live audio — voice agents, live captions — AssemblyAI streams over a secure WebSocket. The current streaming flagship is Universal-3.5 Pro Realtime ($0.45/hr base), which posts a 6.99% word error rate on a real-world voice-agent benchmark and uses neural end-of-turn detection (reading tonality and pacing, ~300ms) instead of raw silence.

The v3 streaming SDK connects with a singular speech_model (streaming has no fallback list) and streams PCM16 mono 16 kHz audio:

# pip install "assemblyai>=1.0.0"
import os
from assemblyai.streaming.v3 import (
    StreamingClient, StreamingClientOptions, StreamingEvents,
    StreamingParameters, TurnEvent,
)

def on_turn(_, event: TurnEvent):
    tag = "FINAL" if event.end_of_turn else "partial"
    print(f"{tag}: {event.transcript}")

client = StreamingClient(StreamingClientOptions(api_key=os.environ["ASSEMBLYAI_API_KEY"]))
client.on(StreamingEvents.Turn, on_turn)
client.connect(StreamingParameters(sample_rate=16000, speech_model="universal-3-5-pro"))
# Feed 16 kHz mono PCM16 chunks (50–1000ms each) via client.stream(chunk)
client.disconnect(terminate=True)  # sends Terminate and closes cleanly

If you'd rather work at the protocol level, connect directly to the v3 WebSocket endpoint. Note the auth header uses your API key with no Bearer prefix, and the old v2 URL (wss://api.assemblyai.com/v2/realtime/ws) is inactive:

import asyncio, json, os, websockets
URL = "wss://streaming.assemblyai.com/v3/ws?sample_rate=16000&speech_model=universal-3-5-pro"

async def run(audio_source):
    async with websockets.connect(URL, additional_headers={"Authorization": os.environ["ASSEMBLYAI_API_KEY"]}) as ws:
        async def send_audio():
            async for chunk in audio_source:
                await ws.send(chunk)
            await ws.send(json.dumps({"type": "Terminate"}))
        async def recv_loop():
            async for raw in ws:
                msg = json.loads(raw)
                if msg["type"] == "Turn":
                    print(("FINAL" if msg["end_of_turn"] else "partial"), msg["transcript"])
                elif msg["type"] == "Termination":
                    return
        await asyncio.gather(send_audio(), recv_loop())

Building a full voice agent

If you're wiring speech-to-text into a full voice agent, you can skip stitching STT, an LLM, and TTS together yourself. AssemblyAI's Voice Agent API runs the entire pipeline over one WebSocket at a flat $4.50/hr — roughly 4x cheaper than OpenAI's Realtime API at about $18/hr. And because it works with no SDK required, it drops cleanly into agentic dev workflows like Claude Code. See how to build with the Voice Agent API for a full walkthrough.

How to choose the right Python speech recognition solution

A survey of 450+ Voice AI builders found 52.5% cited accuracy as their top challenge, and 55% said users' biggest frustration was having to repeat themselves because of transcription errors. Accuracy isn't a nice-to-have. Here's a decision framework:

1. Need offline? Whisper, faster-whisper, Distil-Whisper, or Vosk. Vosk for edge devices; faster-whisper for GPU servers.
2. Need the highest accuracy? Cloud APIs like AssemblyAI's Universal-3.5 Pro outperform open-source on real-world audio, especially proper nouns and entities.
3. Need real-time? AssemblyAI streaming (Universal-3.5 Pro Realtime) or Vosk for offline.
4. Non-English audio? Whisper and AssemblyAI both cover 99+ languages; Vosk covers 20+.
5. Tight budget, low volume? Whisper or faster-whisper on hardware you already own; AssemblyAI's free credit for prototyping.

Comparison table

For a broader look at your options, see the top free speech-to-text APIs and open-source engines, or the complete guide to speech-to-text. Building something latency-sensitive? Start with real-time transcription in Python.

Start now: Try AssemblyAI's API for free — transcription, diarization, and 99+ language support from a single API.

Frequently asked questions

What is the most accurate Python speech-to-text library or API in 2026?

Cloud APIs deliver the highest accuracy on real-world audio. AssemblyAI's Universal-3.5 Pro leads with a 7.69% average code-switching word error rate, outperforming ElevenLabs Scribe v2 and Deepgram Nova-3. Among open-source, Whisper large-v3 is strongest but needs a GPU.

Can Python speech recognition work fully offline without an internet connection?

Yes. Whisper, faster-whisper, Distil-Whisper, and Vosk all run offline. Vosk is lightest for embedded and edge devices, while the Whisper family needs a GPU for practical, near-real-time performance.

How do I transcribe audio in real time with Python?

Use AssemblyAI's v3 streaming WebSocket at wss://streaming.assemblyai.com/v3/ws with a singular speech_model=universal-3-5-pro, streaming 16 kHz PCM16 audio. Universal-3.5 Pro Realtime runs at $0.45/hr base. For offline real-time, Vosk supports live microphone input.

What's the difference between open-source and cloud-based speech recognition in Python?

Open-source runs on your hardware — full control and offline capability, but you own accuracy tuning, scaling, and maintenance. Cloud APIs like AssemblyAI handle infrastructure, deliver higher real-world accuracy, and include diarization, streaming, and redaction, in exchange for per-second cost and connectivity.

Is Whisper better than a cloud speech-to-text API for a Python project?

Whisper excels at offline and multilingual tasks, but cloud APIs win on real-world accuracy, especially proper nouns and noisy audio. Universal-3.5 Pro shows roughly 30% fewer hallucinations than Whisper large-v3 and adds diarization and streaming without GPU management.

How do I build a full voice agent in Python instead of just transcribing?

Rather than stitching STT, an LLM, and TTS together, use AssemblyAI's Voice Agent API — one WebSocket, built on Universal-3.5 Pro Realtime, at a flat $4.50/hr (about 4x cheaper than OpenAI's Realtime API). It also works cleanly with agentic tools like Claude Code.

‍