Universal-3.6
Q4 2026A lower-cost Universal-3.6 streaming model at $0.30/hr, alongside Universal-3.6 Pro.
What We Are Building
What's coming next across the AssemblyAI stack — from transcription and understanding through speech, conversation, and the agent platform — and what shipped in the last 90 days.
Transcribe
Speech-to-text across pre-recorded, realtime, and sync audio — the foundation everything else is built on.
Upcoming
Full-Verbatim Transcription
Q4 2026An option to keep filler words, false starts, and repetitions exactly as spoken, instead of a cleaned-up transcript.
Now available as universal-3-6-pro-preview; GA follows evaluation.
Medical Mode v2
Q4 2026The next version of Medical Mode for clinical transcription.
Veterinary Mode
Q4 2026
Available now in preview via domain: veterinary-v1 (English, Spanish,
French, German). GA in Q4.
Universal-4 Pro
2027The next major accuracy and latency release after Universal-3.6 Pro, across pre-recorded and realtime audio, with the strongest handling of voice-agent audio (noise, interruptions, hesitation, accented speech).
Finance and legal modes, following the Medical Mode pattern already in production — domain vocabulary and formatting tuned per vertical.
Shipped in last 90 days
-
Universal-2 Auto-Routing to Universal-3.5 Pro Universal-2 requests in Chinese, Japanese, Swedish, and Hebrew are now served by Universal-3.5 Pro automatically, at Universal-2 rates.
-
Universal-3.6 Pro (Streaming) The new default streaming model, used when no
speech_modelis sent. 32 languages with native mid-sentence code-switching (14 of them new), entity-aware endpointing, and better handling of background voices. Same $0.45/hr as Universal-3.5 Pro. New languages: Galician, Korean, Catalan, Russian, Afrikaans, Romanian, Cantonese, Marathi, Persian, Estonian, Urdu, Norwegian Nynorsk, Zulu, Xhosa. -
Streaming Speaker Diarization (GA) Speaker labels on streaming audio are generally available, with speakers revised mid-session as more audio arrives. Diarization error rate is about 7%, down from about 10%.
-
Voice Focus Quail 2.2 Voice Focus noise suppression now runs on Quail 2.2, with stronger separation of the primary speaker from secondary talkers.
-
Flexible
language_codeson Universal-3.5 Pro Pre-recordedlanguage_codesaccepts any set of supported languages — English no longer has to be included. -
Speaker Confidence Scores Opt in with
speaker_options.include_speaker_confidenceto get a confidence score for the speaker assigned to each word and utterance. -
Spoken-Form Output Set
format_text: falseon Universal-3.5 Pro pre-recorded audio to get transcripts in spoken form, without written-form formatting. -
Entity-Aware Endpointing Endpointing now detects when a turn ends mid-entity — a phone number, an account ID, a spelled-out name — and waits instead of cutting, with a new end-of-turn confidence field. Turn-taking benchmarks improved 12%, building on the July 22 default-mode improvements (40.7% on phone numbers, 14.4% on emails). On by default.
-
Regional English Localization Language detection can render English transcripts in a regional spelling variant (en_au, en_uk, and more) and return the region-aware language code, without giving up automatic detection or code-switching.
-
Sync STT API Synchronous HTTP transcription: POST a short audio clip and get the finished Universal-3.5 Pro transcript back in the same response, ~134ms median latency. Clips up to two minutes, conversation context for multi-turn voice flows, and a /warm endpoint to keep connection setup off the critical path.
Understand
Turning transcripts into meaning: LLM inference over audio, speaker intelligence, and the guardrails that keep output safe to ship.
Upcoming
LLM Gateway Community Models
Q1–Q4 2026Ongoing catalog expansion across frontier and open-weight models, added as they release.
Audio Input for LLM Gateway
Q4 2026Send audio directly to LLM Gateway models that accept it, alongside text.
Speaker ID Improvements
Q4 2026Speaker identification improvements, including support for multichannel audio.
Voice Fingerprinting
Q1–Q2 2027Recognize the same speaker across recordings, not just within a single file. For meetings, call centers, and cross-session analytics.
Category redaction for entity types you define, building on the static find-and-replace redaction that has already shipped.
Shipped in last 90 days
Models
-
LLM Gateway: 20+ New Models Added since July: GPT-6.1 Sol, Grok 4.7, GPT-6 Sol and Luna, Claude Opus 5.5, DeepSeek V4.1 Flash, the Nemotron 3 family, GLM-5.3, Kimi K3, MiniMax M3, GPT-6 Astra, Claude Opus 5, GPT-5.6 Sol, gemma-4-31b, Gemini 3.7 Flash, Gemini 3.6 Flash, Gemini 3.5 Flash-Lite, Claude Sonnet 5, and GPT-5.6 Luna and Terra. Open-weight models are served through new opt-in providers Together AI, Fireworks AI, and DigitalOcean, and a public
/v1/modelsendpoint lists every model with its pricing. -
qwen3.5-4b-32k-fast New open source model in the LLM Gateway catalog optimized for fast voice AI workloads.
Platform
-
LLM Gateway Get Started Page A Get Started page in the dashboard that takes you from API key to a first LLM Gateway request.
-
LLM Gateway Billing by Provider and Tier LLM Gateway usage is billed and reported per provider and service tier.
-
Zero Data Retention on LLM Gateway LLM Gateway requests can run with zero data retention.
-
Free-Tier LLM Gateway Access Free-tier accounts can call LLM Gateway, so you can try it before adding a card.
Speech Understanding
-
Summarization and Action Items (Open Beta) Chaptered summaries with timestamps, and follow-up tasks with supporting quotes, powered by frontier models — replacing legacy Summarization and Auto Chapters. An effort control handles harder transcripts. Available in US and EU regions.
Speak
Text-to-speech: the voices your application speaks with.
Upcoming
TTS-1
Q4 2026A standalone text-to-speech model for production voice workloads. Low time-to-first-byte, a large voice library, speed control, a pronunciation dictionary, and accurate delivery of phone numbers, emails, and named entities that today’s TTS struggles with.
Voice Cloning
Following TTS-1 launchCustom voices from a short reference clip, built on TTS-1.
Shipped in last 90 days
-
Language-Aware Text Normalization Numbers, dates, and abbreviations are normalized per language across all six supported TTS languages, rather than applying English rules everywhere.
-
Word-Level TTS Timestamps Streaming TTS returns per-word start and end times for generated audio, so a voice agent knows exactly how much of its reply the caller heard before interrupting.
-
In-House TTS Voices Our first AssemblyAI-built TTS voices are live in production behind the Voice Agent API voice catalog, selectable by voice_id in the docs voice reference and the playground.
Interact
Full-duplex conversation — voice agents, dictation, and live translation.
Upcoming
Voice Agent Calls and Logs
Q4 2026A dashboard page to browse your voice agent calls and their logs.
Live Translation
2027Translate conversations in realtime, so each participant hears the call in their own language.
Research toward a native speech-to-speech model for voice agents, aiming for tighter latency, better prosody, and more natural interruption handling — without giving up control over turn-taking.
Shipped in last 90 days
-
Pre-connect Requests Make backend requests while the phone is still ringing, before the call connects. The caller's phone number is included in the request, so you can run a CRM lookup and use the result to decide whether to answer the call at all — or to greet the caller by name with a dynamic, personalized greeting.
-
Voice Agent Webhooks Receive post-call audio, transcripts, and metadata by webhook when a voice agent call ends.
-
Dictation API Real-time dictation with formatting, punctuation, and voice commands built in.
-
SIP Integration Guide Connect an inbound phone number to a voice agent over SIP, replacing the previous Twilio-specific guide.
-
Session Recordings and Transcripts A sessions API returns recordings and transcripts for completed voice agent sessions, so you can surface call history in your own application instead of storing it yourself.
-
Per-Session STT Controls Tune speech recognition per voice agent session: transcription modes for the latency/accuracy trade-off, a contextual transcription prompt, language steering via language_codes, and Voice Focus noise suppression. Agent replies also stream word by word for live captions.
Inference
How models run in production — across transports, regions, and deployment options.
Upcoming
Rate limits that scale automatically by provider, region, and model instead of needing a manual bump, starting in October. Visible token-per-minute limits and automatic failover away from a provider that is down follow.
Self-hosted streaming and sync are available today; November adds Universal-3.6 Pro and standalone speaker diarization. Pre-recorded (async) self-hosting is not planned.
Service Lanes
2027LLM Gateway service tiers (priority, standard, and flex) for trading latency against cost.
On-Device
2027Our models running on the device, for offline and latency-critical workloads where audio never leaves the endpoint.
Shipped in last 90 days
-
Sync API Streaming During Recording Stream audio to the Sync API while the user is still speaking, so transcription starts before the recording ends.
-
Self-Hosted Deploy on Modal Deploy self-hosted sync and streaming transcription to Modal with a single
modal deploy. -
Compressed Audio Input The streaming WebSocket accepts compressed Opus and AAC audio alongside PCM. AAC streams are roughly six times lighter on network I/O than PCM for the same audio.
-
Faster Time-to-Final Final transcripts return roughly twice as fast on forced endpoints — median 427ms to 285ms, P99 899ms to 443ms — with no accuracy cost, via speculative decoding. Now #1 on time-to-first-token on Coval’s public benchmark.
Agent Platform
Building, deploying, and operating voice agents on AssemblyAI.
Upcoming
Voice Agent SDK
Q4 2026Official client libraries that own the connection and audio pipeline, so agents work on the first try. The Python SDK launches in October; TypeScript follows.
Hosted Voice Agents
Q4 2026One-click deploy of Voice Agent SDK agents to AssemblyAI’s cloud, so you don’t run your own agent infrastructure. Live today with an early customer.
Shipped in last 90 days
-
Tool-Parameter Guardrails Voice agent tool calls validate their arguments before running, cutting hallucinated tool arguments from 2.5% to 0% in our evaluations.
-
Voice Agent Starter Repos Starter projects in JavaScript and Python for building a voice agent on AssemblyAI.
Developer Experience
The dashboard, accounts, and tooling that make AssemblyAI easy to adopt — for humans and AI agents.
Upcoming
Voice Standard Bench
Q4 2026Public benchmarks for voice AI, so you can compare models on the workloads you run.
Enhanced Billing Controls
Q1 2027Hard spend caps and team budgets, beyond today’s soft alerts.
Usage and Spend API
Q1 2027Programmatic access to billing and usage data.
Enhanced API Metrics
Q1 2027Deeper dashboard observability: P50 and P95 turnaround time, webhook delivery stats, uptime, and latency histograms.
Shipped in last 90 days
-
Python SDK 1.5 and Node SDK 4.41 The Python SDK adds Dictation and Sync API support; Node SDK 4.41.1 ships alongside.
-
AssemblyAI Agent A coding agent that gets you started with working sample code for your use case. Now the default onboarding flow for all new signups as of 2026-09-25.
-
Self-Serve Billing Details Update your billing address, company name, and tax ID from the dashboard.
-
SSO for Pay-As-You-Go Single sign-on is available to pay-as-you-go accounts as an add-on, not just contract accounts.
-
Python SDK 1.0 One interface across pre-recorded, streaming, and sync transcription, replacing three separate client shapes. Adds SDK coverage for the Sync API, and AsyncTranscriber runs transcriptions concurrently as awaitable coroutines instead of blocking a thread.
-
Dashboard Refresh and Integrated Playground A visual refresh across the dashboard — new palette, typography, and retuned light and dark modes — and the standalone playground now lives inside the dashboard.
-
Single Sign-On (SSO) Self-serve SAML with just-in-time provisioning by email domain, managed from the dashboard. Launched for contract accounts; pay-as-you-go added 2026-08-14.