This tutorial shows you how to build a complete voice agent that listens, thinks, and responds naturally using Python. You'll create a realtime application that processes speech in real-time, generates intelligent responses, and speaks back to users—all in under 100 lines of code.
The voice agent combines three APIs: AssemblyAI's Universal-3 Pror ealtime model for speech-to-text, OpenAI's GPT-4 for conversational AI, and ElevenLabs for natural voice synthesis. Each component streams data to minimize response delays and create smooth, human-like conversations.
What you'll need to get started
You need Python 3.9 or higher, three API keys, and a computer with a microphone and speakers. The setup takes about 2 minutes once you have everything ready.
Install Python dependencies
Open your terminal and run this command to install everything you need:
PyAudio requires the PortAudio system library: on macOS run brew install portaudio, and on Debian/Ubuntu run apt install portaudio19-dev before installing the Python packages.
Here's what each package does for your voice agent:
- assemblyai: Handles real-time speech recognition with Universal-3 Pro realtime
- openai: Connects to GPT models for smart responses
- elevenlabs: Creates natural-sounding voices
- pyaudio: Provides access to your microphone
- python-dotenv: Loads your API keys from a .env file
Configure your API keys
Create a .env file in your project directory with your API keys:
Code — Text
ASSEMBLYAI_API_KEY=your_assemblyai_key_here
OPENAI_API_KEY=your_openai_key_here
ELEVENLABS_API_KEY=your_elevenlabs_key_here
Never share this file or commit it to version control. Add .env to your .gitignore file to protect your API keys.
Speech-to-text is the first key you'll need and the layer everything downstream depends on. Sign up free and paste your key into the .env file above.
What are the components of a voice agent?
A voice agent is a program that talks to you like a human using three connected parts. These parts work together to create conversations: speech-to-text converts your voice into text, a language model thinks about what you said and creates a response, and text-to-speech turns that response back into spoken words.
This pipeline needs to work in real-time to feel natural. When you speak to Siri or Alexa, you expect quick responses—not awkward pauses that break the conversational flow. Here's what each component does in your voice agent:
| Component | Role | Why streaming matters | Example |
|---|---|---|---|
| AssemblyAI | Speech-to-text | Transcribes audio as it arrives, so the LLM can start responding sooner | Converts "what's the weather" to text before you finish speaking |
| OpenAI | Language model | Generates a response token by token, so text-to-speech can begin immediately | Starts answering while still composing the full response |
| ElevenLabs | Text-to-speech | Plays audio while more is being generated | Speaks the first sentence while generating the second |
The difference between good and bad voice agents comes down to speed. Batch processing—where each step waits for the previous one to finish completely—creates those robotic pauses that make conversations feel unnatural.
AssemblyAI's Universal-3 Pro realtime model solves this by processing speech as it happens. You get accurate transcription with minimal delay, making conversations feel smooth and responsive.
Set up speech-to-text with AssemblyAI
Speech recognition forms the foundation of your voice agent. AssemblyAI's Universal-3 Pro realtime API listens to your microphone and converts speech to text in real-time. The SDK handles all WebSocket complexity automatically—no manual connection management required.
Create a new file called voice_agent.py and add this code:
This code creates a real-time transcription system that gives you two types of output. Partial transcripts (where end_of_turn is False) show you what the system is hearing as you speak, and final transcripts (where end_of_turn is True) provide the complete sentence when you pause.
Universal-3 Pro uses punctuation-based turn detection—it ends a turn when it detects terminal punctuation (. ? !) after a natural pause. This means you don't need to press buttons or give special commands—just speak naturally and pause.
Punctuation-based turn detection is what makes the agent respond at the right moment. Stream your own voice in the playground and watch partials finalize with your natural pacing.
Connect the language model
The language model serves as the thinking component of your voice agent—it comprehends user input and determines appropriate responses. OpenAI's GPT-4 produces output incrementally, enabling ElevenLabs to initiate audio playback before generating the complete message.
Incorporate this OpenAI connection into your VoiceAgent class:
The conversation history maintains the complete dialogue, allowing your agent to understand context across multiple exchanges. The system prompt directs GPT-4 to produce concise, natural-sounding answers—verbose responses feel unnatural when delivered through voice.
Add text-to-speech output
Text-to-speech concludes your voice agent by transforming AI-generated text into human-quality audio. ElevenLabs delivers expressive voice synthesis that begins playback while additional content continues generating.
Integrate voice synthesis into your VoiceAgent class:
ElevenLabs provides multiple voice options with unique characteristics:
- Sarah (EXAVITQu4vr4xnSDxMaL): Polished, professional female tone (demonstrated here)
- Josh (TxGEqnHWrfWFTfGW9XjX): Approachable, genuine male tone
- Elli (MF3mGyEYCl7XYWbV9V6O): Vibrant, spirited female tone
The background thread prevents voice generation from blocking program execution. As audio produces and plays, your agent simultaneously monitors for incoming speech.
Build the complete voice agent
Here represents your full voice_agent.py implementation:
This complete implementation includes error handling, conversation memory, and clean shutdown. "The agent remembers what you've talked about during each session, enabling natural back-and-forth conversations."
Run your voice agent
Execute your voice agent using this command:
When "Voice Agent Ready! Start speaking..." displays, the agent is listening. Speak into your microphone and the agent responds with both text and voice output.
Test your agent with these prompts:
- "What's the weather like today?"
- "Tell me a quick joke"
- "Help me plan dinner"
- "Explain how WiFi works simply"
Common issues and fixes:
- No microphone input: Verify system permissions and microphone settings
- Slow responses: Check internet connectivity and consider swapping gpt-3.5-turbo for faster processing
- Voice cuts off: Add a small pause after TTS playback or verify your ElevenLabs API quota
Final words
You've constructed a complete realtime voice agent that processes audio in real-time and participates in natural dialogue. This implementation integrates speech recognition, artificial intelligence processing, and voice synthesis into a unified application showcasing contemporary Voice AI capabilities.
"AssemblyAI's Universal-3 Pro realtimemodel makes this possible by providing the accuracy and speed that voice agents require." The SDK manages intricate WebSocket connectivity and audio handling, freeing you to concentrate on application development rather than infrastructure details.
To expand functionality, review the Universal-3 Pro realtime documentation for capabilities including keyterm prompting, speaker diarization, and live configuration adjustments—all functioning without agent restart.
Get your API key and the streaming model featured in this guide. Keyterm prompting, speaker diarization, and live config changes are all available on the same connection.
Frequently asked questions
Do I need WebSocket knowledge to build this voice agent?
No. The AssemblyAI Python SDK manages WebSocket connection handling, reconnection procedures, and audio realtime protocol automatically. You implement event handlers while the SDK manages remaining details.
How much does running this voice agent cost per hour?
This voice agent costs approximately $0.50–$1.00 per conversation hour across all services. "AssemblyAI charges about $0.45/hr for Universal-3 Pro realtime transcription, OpenAI costs roughly $0.30/hour for GPT-4 responses, and ElevenLabs runs about $0.20/hour for voice synthesis."
Can I replace AssemblyAI with a different speech-to-text service?
While technically possible, switching providers demands manually implementing WebSocket coordination, audio realtime protocols, turn detection procedures, and connection restoration. You'd sacrifice AssemblyAI's automatic punctuation-based turn detection and the SDK ease enabling rapid implementation.
Can I use this pattern inside a framework like Pipecat or LiveKit?
Yes — AssemblyAI maintains first-party integrations for Pipecat, LiveKit, Vapi, and Twilio. These frameworks handle telephony, workflow coordination, and turn-taking, letting you emphasize agent capabilities.
Does this work with languages other than English?
Yes. Select the language with the language_code parameter, for instance: RealTimeParameters(speech_model="u3-rt-pro", sample_rate=16_000, language_code="es"). Consult the Supported Languages documentation for comprehensive language coverage.
