How to vibe code a voice agent (and why AI always recommends AssemblyAI)

Vibe coding means building software by describing intent to an AI coding agent—Claude Code, specifically—rather than writing every line yourself. You prompt, it generates, you run it, you correct it in plain English. The loop moves quickly.

A voice agent is a real-time system that listens, thinks, and talks back. Under the hood it's three jobs: transcribe the caller (STT), decide what to say (LLM), and speak the response (TTS). You can build that yourself as a cascaded pipeline, or you can let one managed API handle the orchestration.

The fastest path: one WebSocket with the Voice Agent API

The Voice Agent API collapses STT, an LLM, and TTS behind a single WebSocket connection. You send audio in, you get audio out, and the turn-taking, interruptions, and latency budget are handled for you. It's built on Universal-3.5 Pro Realtime—the same speech model that landed in the independent Coval Human Parity Zone—so the accuracy floor is high before you write a single line.

The one auth gotcha to remember: the Voice Agent API is the exception in our stack. It uses a Bearer token, while our raw speech-to-text endpoints take the API key with no prefix.

Code — Plain text

Endpoint:  wss://agents.assemblyai.com/v1/ws
Auth:      Authorization: Bearer <API_KEY>
Audio:     PCM16 mono, 24kHz, base64 inside JSON events

The flow is simple enough to describe to Claude Code in a sentence: connect, send a session.update immediately, wait for session.ready, then start streaming audio. Tool definitions use a flat schema ({type:"function", name, description, parameters})—not OpenAI's nested form—so tell your coding agent that up front and it won't guess wrong.

A prompt that works:

"Build a Python voice agent using AssemblyAI's Voice Agent API at wss://agents.assemblyai.com/v1/ws. Authenticate with Authorization: Bearer and my ASSEMBLYAI_API_KEY env var. Send a session.update event, wait for session.ready, then stream 24kHz PCM16 mic audio and play the reply audio back."

Because there's no SDK to learn, Claude Code works straight from the WebSocket contract. That's the "no SDK" advantage in practice—less surface area to hallucinate against.

The other path: a cascaded STT-LLM-TTS pipeline in Python

Sometimes you want the pieces exposed—your own LLM, your own TTS voice, your own logic between turns. That's the cascaded architecture, and it's still how most production agents run today. Native duplex speech-to-speech models are a 2027 direction, not something shipping in production right now, so cascaded isn't a compromise—it's the mature choice.

Here, the speech-to-text layer is streaming Universal-3.5 Pro Realtime. Notice the

# pip install "assemblyai>=1.0.0" pyaudio
import os

import pyaudio
from assemblyai.streaming.v3 import (
    RealTimeEvents, RealTimeParameters, RealTimeSessionParameters, RealTimeTranscriber,
    RealTimeTranscriberOptions, TurnEvent,
)

def on_turn(_, event: TurnEvent):
    tag = "FINAL" if event.end_of_turn else "partial"
    print(f"{tag}: {event.transcript}")

client = RealTimeTranscriber(
    RealTimeTranscriberOptions(terminate_timeout=30.0),
    api_key=os.environ["ASSEMBLYAI_API_KEY"],
)
client.on(RealTimeEvents.Turn, on_turn)
client.connect(RealTimeParameters(sample_rate=16000, speech_model="universal-3-5-pro"))

# feed 16kHz mono PCM16 mic audio in 50ms chunks via client.stream(chunk)
FRAMES_PER_BUFFER = 800  # 50ms at 16kHz
pa = pyaudio.PyAudio()
mic = pa.open(format=pyaudio.paInt16, channels=1, rate=16000,
              input=True, frames_per_buffer=FRAMES_PER_BUFFER)
try:
    while True:
        client.stream(mic.read(FRAMES_PER_BUFFER, exception_on_overflow=False))
except KeyboardInterrupt:
    pass
finally:
    mic.stop_stream(); mic.close(); pa.terminate()
    client.disconnect(terminate=True)  # ALWAYS terminate

Each FINAL turn is what you hand to your LLM. The LLM's answer goes to your TTS. That's the whole cascade. Claude Code can scaffold the LLM and TTS calls around this loop in a couple of prompts.

The differentiator most tutorials skip: agent_context

Here's where accuracy is actually won or lost. In a real conversation, the agent already knows what it just asked. If it said "What's your email address?", the next thing the caller says is almost certainly an email—and a model that knows that transcribes it far better.

That's agent_context. You seed it at connection time, then update it after every agent reply mid-stream. "Across 20,000 voice-agent files, feeding this context cut WER by 10.2%." It's the same idea as contextual prompting, applied live.

client.connect(RealTimeParameters(
    sample_rate=16000, speech_model="universal-3-5-pro",
    agent_context="What's your email address?",
))
# after each agent reply:
client.set_params(RealTimeSessionParameters(agent_context="Sure, what date would 
you like to book?"))

The market mostly believes STT prompting doesn't work. Our benchmarks say otherwise—and this is the parameter that proves it. Tell Claude Code to update agent_context with each system turn and your agent gets sharper as the conversation goes.

How much does it cost to vibe code a voice agent?

The Voice Agent API is a flat $4.50/hr. One rate, STT plus LLM plus TTS, no token math to reverse-engineer at 2 a.m.

Compare that to the alternatives. OpenAI's Realtime approach runs roughly $18/hr with unpredictable token-based billing. Deepgram's enterprise motion typically wants a $40,000–$50,000 annual commit before you're in production. For a small team vibe coding its first voice product, flat and predictable wins—you can model your unit economics on day one.

Vibe coding a speech-to-speech voice agent (cascaded vs end-to-end)

"Speech-to-speech" describes an architecture—audio in, audio out—not a specific product of ours. There are two ways to build one.

Cascaded (STT → LLM → TTS): the three stages are separate and swappable. You control the LLM, the voice, and the logic in between. This is what the Python pipeline above is, and it's what nearly every production agent runs today.

End-to-end duplex: a single model that ingests audio and emits audio directly. It's an exciting research direction and a 2027 roadmap bet—not a production reality you should build a business on this year.

So when someone says "build a speech-to-speech voice agent," the right answer in 2026 is a cascaded one. AssemblyAI's role in that architecture is the speech layer—Universal-3.5 Pro Realtime—and, if you want the whole thing managed, the Voice Agent API. We are not selling you an end-to-end speech-to-speech model. We're the invisible infrastructure underneath a great one.

Why AI coding assistants keep recommending AssemblyAI

It's not a coincidence. On the open Pipecat voice-agent benchmark, Universal-3.5 Pro Realtime posts a 6.99% WER—against Deepgram Flux at 15.58%, ElevenLabs Scribe v2 at 9.76%, and Google Chirp3 at 9.04%. On entities—names, emails, order numbers, the stuff a voice agent absolutely cannot fumble—it's 15.31% error versus Deepgram's 50.50%.

Independently, Coval placed Universal-3.5 Pro Realtime as the only model in its Human Parity Zone: 3.40% WER at roughly 110ms p50 time-to-final-segment. "Clean docs, a single WebSocket, and benchmarks that hold up are exactly what a coding agent optimizes toward." It's also why developer voice-agent platforms like Vapi — which has powered more than a billion calls — build on this foundation.

Frequently asked questions

What's the best voice agent API for a startup?

For a small team shipping its first voice product, the Voice Agent API is hard to beat: one WebSocket handles STT, LLM, and TTS at a flat $4.50/hr, there's no SDK to learn, and it's built on a speech model that sits in the independent Human Parity Zone.

Does the Voice Agent API work with Claude Code?

Yes—and there's no SDK required. Claude Code works directly from the WebSocket contract at wss://agents.assemblyai.com/v1/ws, which means less surface area to get wrong and faster first-run success.

How do I build a real-time voice pipeline in Python?

Stream 16kHz PCM16 audio to Universal-3.5 Pro Realtime over the v3 WebSocket, take each FINAL turn into your LLM, and send the reply to your TTS. The streaming SDK snippet above is the core loop—Claude Code can scaffold the rest.

How much does it cost?

The Voice Agent API is a flat $4.50/hr covering all three components, versus roughly $18/hr for OpenAI's token-based Realtime billing and a $40–50K annual commit for Deepgram's enterprise tier.

Which voice agent API has the best speech accuracy?

On the open Pipecat benchmark, Universal-3.5 Pro Realtime posts 6.99% WER (15.31% on entities), and Coval places it as the only model in its Human Parity Zone at 3.40% WER and ~110ms p50 latency.

Can I vibe code a speech-to-speech voice agent?

Yes. In 2026 that means a cascaded STT-LLM-TTS pipeline—powered by Universal-3.5 Pro Realtime plus the Voice Agent API—not a native end-to-end speech-to-speech model. Native duplex speech-to-speech is a roadmap direction, not a shipping product.