Vapi voice agent with AssemblyAI Universal-3.5 Pro Realtime

Vapi is one of the fastest ways to ship a production voice agent — it handles telephony, turn-taking, and orchestration so you don't have to, and it supports 14+ speech-to-text providers. This guide shows you how to build a Vapi voice agent on AssemblyAI's Universal-3.5 Pro Realtime model: you bring your AssemblyAI key, Vapi handles the rest.

Why Universal-3.5 Pro Realtime for a Vapi voice agent?

The transcriber is the part of a voice agent that decides what your LLM actually responds to — get it wrong and everything downstream is wrong. Universal-3.5 Pro Realtime is AssemblyAI's flagship real-time model: it leads Pipecat's open real-agent benchmark at 6.99% WER, streams partial and final transcripts around 150 ms P50, and is tuned for the emails, phone numbers, and account numbers that come up on every real call. Selecting it in Vapi is a one-line change.

Heads up on model IDs: the older u3-rt-pro auto-routes to universal-3-5-pro on August 7, 2026 and stops accepting the old ID around September 25, 2026. Use universal-3-5-pro going forward.

Setup: add AssemblyAI to Vapi

Step 1 — Add your API key

1. Go to dashboard.vapi.ai
2. Navigate to Settings → Transcriber Providers
3. Add your AssemblyAI API key

Step 2 — Create an assistant (dashboard)

1. Click Create Assistant
2. Under Transcriber, select Assembly AI
3. Under Model, select universal-3-5-pro (Universal-3.5 Pro Realtime)
4. Save and test via the web call button

Step 3 — Create an assistant (API)

python create_assistant.py create

This creates a fully configured assistant:

{
  "transcriber": {
    "provider": "assembly-ai",
    "model": "universal-3-5-pro",
    "language": "en",
    "keytermsPrompt": ["YourBrand", "SpecialTerm"],
    "confidenceThreshold": 0.4
  }
}

Quick start

git clone https://github.com/kelsey-aai/voice-agent-vapi-assemblyai
cd voice-agent-vapi-assemblyai

pip install -r requirements.txt
cp .env.example .env
# Edit .env with your keys

# Create an assistant
python create_assistant.py create

# Make an outbound test call (requires Twilio number in .env)
python create_assistant.py call --assistant-id <id> --phone +1XXXXXXXXXX

# Start the webhook server
uvicorn webhook_server:app --port 8000

Keyterm prompting

This is one of the biggest accuracy levers available in Vapi. Boost recognition for domain-specific vocabulary a general speech model would otherwise miss:

"keytermsPrompt": [
    "hemoglobin A1c",     // medical
    "HIPAA",              // compliance term
    "Jardiance",          // drug name
    "deductible"          // insurance
]

Up to 100 keyterms, each up to 50 characters. Takes effect immediately on the next call — no assistant restart needed. Keyterm prompting is included on Universal-3.5 Pro Realtime.

Supported languages

Universal-3.5 Pro Realtime supports 18 languages — including English, Spanish, French, German, Italian, and Portuguese — with native mid-sentence code-switching. Set the language in the transcriber config:

{ "transcriber": { "provider": "assembly-ai", "model": "universal-3-5-pro", "language": "es" } }

When to choose AssemblyAI over Deepgram in Vapi

Resources

• Vapi AssemblyAI provider docs
• Universal-3.5 Pro Realtime streaming docs
• Vapi API reference

Frequently asked questions

What is the best transcriber for a Vapi voice agent?

For most production Vapi voice agents, AssemblyAI's Universal-3.5 Pro Realtime is the strongest transcriber choice: it leads Pipecat's open real-agent benchmark at 6.99% WER, streams transcripts around 150 ms P50, includes keyterm prompting, and adds Context Carryover so the model transcribes each answer in the context of your agent's question. You select it in Vapi by choosing Assembly AI as the transcriber and setting the model to universal-3-5-pro.

How do I use AssemblyAI in Vapi?

Add your AssemblyAI API key under Settings → Transcriber Providers, then on any assistant choose Assembly AI as the transcriber and select the universal-3-5-pro model — or set "transcriber": {"provider": "assembly-ai", "model": "universal-3-5-pro"} via the API. From there you can add keytermsPrompt for custom vocabulary and set language for non-English calls.

AssemblyAI vs Deepgram in Vapi — which should I use?

Choose AssemblyAI Universal-3.5 Pro Realtime when accuracy on real conversations matters — it posts 6.99% WER on Pipecat's open benchmark and is strong on the alphanumeric entities (account numbers, emails, serial codes) that voice agents mishear most. It also brings included keyterm prompting and Context Carryover. Both providers stream at low latency in Vapi; AssemblyAI's edge is transcript accuracy on the inputs that actually break agents.

Does Vapi support AssemblyAI's Universal-3.5 Pro Realtime model?

Yes. Select Assembly AI as your Vapi transcriber and set the model to universal-3-5-pro. (The older u3-rt-pro identifier still works but is being retired — it auto-routes to universal-3-5-pro on August 7, 2026 — so set the current string.)

How much does a Vapi voice agent with AssemblyAI cost?

AssemblyAI's Universal-3.5 Pro Realtime is $0.45/hr for transcription, with keyterm prompting included and no minimum commitment. Vapi's platform fee and your LLM/TTS providers are billed separately. AssemblyAI has a free tier so you can wire up and test your Vapi assistant before adding a card.