Build a real-time AI voice bot using Python, AssemblyAI, and OpenAI

You can build a real-time AI voice bot in Python by chaining three components: a streaming speech-to-text model that transcribes the caller, an LLM that generates a reply, and a text-to-speech model that speaks it back. This tutorial builds an AI-powered dental clinic receptionist using AssemblyAI's Universal-3.6 Pro Realtime for speech-to-text, OpenAI for responses, and AssemblyAI's Text-to-Speech API for voice synthesis, so the bot listens and speaks through one SDK and one API key.

AI voice bots are changing how businesses handle customer interactions—and for developers, they're a chance to build intelligent, scalable systems that make those interactions faster and more natural. The speech-to-text layer is the foundation: if the bot mishears a name, a phone number, or a date, everything downstream breaks. That's why we use Universal-3.6 Pro Realtime, AssemblyAI's flagship real-time model, which interprets each turn in the context of the conversation and lets you tune the latency-versus-accuracy trade-off per use case.

The steps below build the complete program in a single file, main.py.

Key components of the AI voice bot

An AI voice bot has three major components working in sequence:

• Universal-3.6 Pro Realtime speech-to-text: AssemblyAI's flagship real-time model transcribes the caller's speech with low latency and strong accuracy on names and numbers. Context carryover means each turn is interpreted in the context of prior turns—useful when a caller references something they said earlier. (New to ASR? Start with what is automatic speech recognition.)
• Natural language processing: OpenAI's language models generate context-aware responses.
• Voice synthesis: AssemblyAI's Text-to-Speech API speaks the response back to the caller, closing the conversational loop. Its model, AssemblyAI-TTS-1, is built for conversational voice agents.

The steps below show how to build the bot, with code snippets for each component.

Step 1: Install required Python libraries

Run the following in your terminal to install the dependencies:

brew install portaudio
pip install "assemblyai>=1.8.2" pyaudio sounddevice openai python-dotenv

These power the core functions: streaming transcription and speech synthesis (assemblyai), microphone capture (PyAudio), speaker playback (sounddevice), response generation (openai), and secure key loading (python-dotenv). The Text-to-Speech client ships in the assemblyai package from version 1.8.2, so one SDK covers both directions of the conversation. PortAudio is the system library that PyAudio and sounddevice use for audio input and output; on Debian/Ubuntu, install it with apt install portaudio19-dev.

Step 2: Set up credentials securely

Never hardcode API keys in your source. Create a .env file in your project directory and add your keys there:

ASSEMBLYAI_API_KEY=your_assemblyai_key_here
OPENAI_API_KEY=your_openai_key_here

The same AssemblyAI key covers both Universal-3.6 Pro Realtime and the Text-to-Speech API, so the bot needs only two keys.

Get your free API key from AssemblyAI and set up billing to access Universal-3.6 Pro Realtime. As a first-time user, you'll get $50 in free credits that work for both async and real-time transcription, Speech Understanding, and more.

Create a file called main.py, load the keys from the environment, and set up the assistant class:

import os
import logging
from typing import Type

from dotenv import load_dotenv
import pyaudio
import sounddevice as sd
import assemblyai as aai
from assemblyai.streaming.v3 import (
    BeginEvent,
    RealTimeError,
    RealTimeEvents,
    RealTimeParameters,
    RealTimeTranscriber,
    RealTimeTranscriberOptions,
    TerminationEvent,
    TurnEvent,
)
from openai import OpenAI

load_dotenv()

# Configuration
DEBUG_MODE = False  # Set to True to see all logs and debug messages
FRAMES_PER_BUFFER = 800  # 50ms of audio at 16kHz
TTS_SAMPLE_RATE = 24000  # Text-to-Speech output: 24 kHz, 16-bit mono PCM

if DEBUG_MODE:
    logging.basicConfig(level=logging.INFO)
else:
    logging.basicConfig(level=logging.CRITICAL)
    for noisy in ("httpx", "httpcore", "openai"):
        logging.getLogger(noisy).setLevel(logging.CRITICAL)

logger = logging.getLogger(__name__)

class AI_Assistant:
    def __init__(self):
        # Load keys from environment — never hardcode them
        self.assemblyai_api_key = os.getenv("ASSEMBLYAI_API_KEY")
        self.openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

        # Text-to-Speech settings, reused for every reply
        self.tts_config = aai.TTSConfig(
            voice_id="49c443a4-8a1d-5f7e-b70b-98bdbd9823c3",  # Audrey
            language="english",
            sample_rate=TTS_SAMPLE_RATE,
        )

        self.client = None
        self.microphone_stream = None
        self.pyaudio_instance = None
        self.is_streaming = False

        # Conversation prompt
        self.full_transcript = [
            {"role": "system", "content": "You are a receptionist at a dental clinic. Be resourceful and efficient."},
        ]

        # Track conversation state for latency optimization
        self.is_processing = False
        self.running_transcript = ""   # Accumulates finalized transcripts
        self.latest_partial = ""       # Current partial transcript
        self.should_process_on_next_final = False

        global ai_assistant_instance
        ai_assistant_instance = self

Step 3: Universal-3.6 Pro Realtime transcription with AssemblyAI

At the core of the bot is Universal-3.6 Pro Realtime. AssemblyAI's RealTimeTranscriber handles the audio stream and emits events during the session. The client connects to the current streaming endpoint, wss://streaming.assemblyai.com/v3/ws, automatically.

Create a start_transcription method to build the client, register event handlers, and stream from the microphone with PyAudio:

def start_transcription(self):
    self.client = RealTimeTranscriber(
        RealTimeTranscriberOptions(),
        api_key=self.assemblyai_api_key,
    )

    self.client.on(RealTimeEvents.Begin, on_begin)
    self.client.on(RealTimeEvents.Turn, on_turn)
    self.client.on(RealTimeEvents.Termination, on_terminated)
    self.client.on(RealTimeEvents.Error, on_error)

    self.client.connect(
        RealTimeParameters(
            sample_rate=16000,
            speech_model="universal-3-6-pro",
        )
    )

    self.pyaudio_instance = pyaudio.PyAudio()
    self.microphone_stream = self.pyaudio_instance.open(
        format=pyaudio.paInt16,
        channels=1,
        rate=16000,
        input=True,
        frames_per_buffer=FRAMES_PER_BUFFER,
    )

    self.is_streaming = True
    try:
        while self.is_streaming:
            self.client.stream(
                self.microphone_stream.read(FRAMES_PER_BUFFER, exception_on_overflow=False)
            )
    except OSError:
        pass  # Microphone stream was closed while stopping

Add a stop_transcription method to cleanly disconnect and release the microphone. We stop transcription while the bot generates a response, then restart it when ready to listen again—a simple approach that avoids complex interrupt handling for this tutorial:

def stop_transcription(self):
    self.is_streaming = False
    if self.client:
        self.client.disconnect(terminate=True)
        self.client = None
    if self.microphone_stream:
        self.microphone_stream.stop_stream()
        self.microphone_stream.close()
        self.microphone_stream = None
    if self.pyaudio_instance:
        self.pyaudio_instance.terminate()
        self.pyaudio_instance = None

With Universal-3.6 Pro Realtime, the client returns immutable transcripts via Turn objects. Each Turn has a transcript field with finalized words, an end_of_turn boolean, and metadata like end_of_turn_confidence. The on_turn handler accumulates text and processes it when end_of_turn is true—sending to the LLM immediately for the lowest latency:

ai_assistant_instance = None

def on_turn(client: Type[RealTimeTranscriber], event: TurnEvent):
    if not event.transcript or ai_assistant_instance.is_processing:
        return

    ai_assistant_instance.latest_partial = event.transcript
    print(f"\r{event.transcript}", end="", flush=True)

    if event.end_of_turn:
        if ai_assistant_instance.should_process_on_next_final:
            # Final for a partial we already processed — ignore it
            ai_assistant_instance.should_process_on_next_final = False
            ai_assistant_instance.latest_partial = ""
        else:
            ai_assistant_instance.should_process_on_next_final = True
            ai_assistant_instance.process_turn()

Add a process_turn method that combines finalized text with the latest partial and sends the complete text to the LLM:

def process_turn(self):
    complete_text = self.running_transcript
    if self.latest_partial:
        complete_text = f"{complete_text} {self.latest_partial}".strip()

    self.running_transcript = ""

    if complete_text.strip():
        self.generate_ai_response(complete_text)

Add the remaining event handlers outside the class:

def on_begin(client: Type[RealTimeTranscriber], event: BeginEvent):
    if DEBUG_MODE:
        logger.info(f"Session started: {event.id}")
    print("\n[Listening... Start speaking]")

def on_terminated(client: Type[RealTimeTranscriber], event: TerminationEvent):
    if DEBUG_MODE:
        logger.info(f"Session terminated: {event.audio_duration_seconds}s processed")

def on_error(client: Type[RealTimeTranscriber], error: RealTimeError):
    if DEBUG_MODE:
        logger.error(f"Streaming error: {error}")

Step 4: Generate responses with OpenAI

Once transcription is wired up, use OpenAI to generate context-aware responses. Add this method to the AI_Assistant class:

def generate_ai_response(self, transcript_text):
    self.is_processing = True
    self.stop_transcription()

    self.full_transcript.append({"role": "user", "content": transcript_text})
    print(f"\nPatient: {transcript_text}\n")

    stream = self.openai_client.chat.completions.create(
        model="gpt-4.1-mini",
        messages=self.full_transcript,
        stream=True,
    )

    print("AI Receptionist: ", end="", flush=True)
    ai_response = ""
    for chunk in stream:
        if chunk.choices[0].delta.content:
            content = chunk.choices[0].delta.content
            ai_response += content
            print(content, end="", flush=True)
    print()

    self.generate_audio(ai_response)

    self.is_processing = False
    self.running_transcript = ""
    self.latest_partial = ""
    self.should_process_on_next_final = False
    self.start_transcription()

This stops transcription to prevent interruptions, appends the user's message, streams OpenAI's reply token by token, passes the full response to the Text-to-Speech step, then resets state and restarts listening.

Step 5: Voice synthesis with AssemblyAI Text-to-Speech

The last link in the chain turns OpenAI's reply into speech. AssemblyAI's Text-to-Speech API runs AssemblyAI-TTS-1, a model trained on millions of hours of audio, including real conversational data, and built for voice agents. It delivers frontier-level quality, runs faster and costs less than other production voice agent TTS models, and has unlimited concurrency. The Python SDK's TTSSession opens a WebSocket session, and its speak() method sends one text and yields the audio as it's generated, so playback starts before the whole reply has been synthesized.

Add this method to the AI_Assistant class:

def generate_audio(self, text):
    self.full_transcript.append({"role": "assistant", "content": text})

    # One session per reply: the bot stops listening while it speaks,
    # so no connection has to sit idle between turns.
    with aai.TTSSession(self.tts_config, api_key=self.assemblyai_api_key) as session:
        with sd.RawOutputStream(
            samplerate=TTS_SAMPLE_RATE, channels=1, dtype="int16"
        ) as speaker:
            for chunk in session.speak(text):  # raw 24 kHz 16-bit mono PCM
                speaker.write(chunk)  # plays while the rest is still arriving

A few details make this work end to end:

• Raw PCM straight to the speakers. The session returns raw 24 kHz, 16-bit mono PCM with no header, which is exactly what sounddevice's RawOutputStream plays. speaker.write() blocks until there's room in the device buffer, so playback paces itself.
• No echo back into the microphone. Leaving the RawOutputStream block waits for the last audio to finish playing, so generate_audio returns only after the bot has stopped talking. Transcription restarts after that, and the bot doesn't hear itself.
• An explicit voice. The config sets voice_id to Audrey, one of the thousands of voices in the voice library. Swap in any voice ID from the library, or set language to Spanish, French, German, Italian or Portuguese: every voice speaks all six languages.
• A session per reply. Because this bot stops listening while it talks and restarts afterward, opening a short session for each reply keeps the code simple. A production agent usually holds one session open for the whole call and sends each turn as a request on it; the Text-to-Speech quickstart shows that pattern.

Test streaming accuracy on your own audio. See how Universal-3.6 Pro Realtime handles names, phone numbers, and accents in the playground before you wire it into your bot.

Step 6: Finalize the AI voice bot

Add the entry point to the end of main.py, outside the AI_Assistant class:

if __name__ == "__main__":
    greeting = "Thank you for calling Vancouver dental clinic. My name is Sonny, how may I assist you?"
    print(f"AI Receptionist: {greeting}")

    ai_assistant = AI_Assistant()
    ai_assistant.generate_audio(greeting)

    try:
        ai_assistant.start_transcription()
        input()  # Keep the program running until interrupted
    except KeyboardInterrupt:
        if DEBUG_MODE:
            print("\nStopping...")
    finally:
        ai_assistant.stop_transcription()

You'll hear the bot's greeting, then respond by speaking—and the conversation begins.

Final words

By combining AssemblyAI's Universal-3.6 Pro Realtime, OpenAI, and AssemblyAI's Text-to-Speech API, you've built a voice bot that handles real-time conversation end to end. It's a strong starting point for call centers, customer support, and virtual receptionists where human-like interaction matters.

From here, the listening layer is the part that makes or breaks the experience. Two levers are worth tuning first: pick the right latency mode for your traffic (min latency for a snappy agent, max accuracy for noisy ordering), and add a keyterms_prompt for the names and terms your bot hears most. The real-time transcription in Python guide plus the streaming docs cover more advanced configuration. To go deeper, see our overview of AI voice agents.

On the speaking side, the next step is to start talking before the LLM finishes. TTSSession also exposes generate(), flush() and audio(), so you can send the reply one sentence at a time as OpenAI streams it. Optimizing accuracy and latency covers that pattern, and choosing an interface compares the WebSocket session with the HTTP endpoint.

Build your voice agent today. Get your free API key and start with Universal-3.6 Pro Realtime for speech-to-text and AssemblyAI's Text-to-Speech API for voice output.

Frequently asked questions

How do I build a real-time AI voice bot in Python?

You build a real-time voice bot by chaining three components: a streaming speech-to-text model transcribes microphone audio, an LLM generates a response, and a text-to-speech model speaks it back. In Python, connect AssemblyAI's RealTimeTranscriber to a microphone over wss://streaming.assemblyai.com/v3/ws, handle TurnEvent callbacks to detect end of turn, pass the transcript to OpenAI, and speak the reply with AssemblyAI's Text-to-Speech API through a TTSSession.

What is the best speech-to-text model for a real-time voice bot?

AssemblyAI's Universal-3.6 Pro Realtime is purpose-built for real-time voice applications. It delivers immutable, low-latency transcripts with strong accuracy on names, phone numbers, and email addresses, and its context carryover interprets each turn in the context of the conversation. Those structured entities are exactly what a voice bot acts on, which makes entity accuracy more important than average word accuracy alone.

How do I keep my API keys secure in a Python voice bot?

Store API keys in a .env file and load them with python-dotenv rather than hardcoding them in your source. This keeps secrets out of version control and lets you use different keys across environments—the script in this tutorial reads ASSEMBLYAI_API_KEY and OPENAI_API_KEY from the environment. One AssemblyAI key covers both speech-to-text and text-to-speech.

How do I reduce latency in a real-time voice bot?

Send the accumulated transcript to the LLM immediately on end_of_turn rather than waiting for formatted text. Universal-3.6 Pro Realtime also offers configurable modes—choose min latency for the fastest possible response in a conversational agent, or balanced for general-purpose traffic.

How much does it cost to build a voice bot with AssemblyAI?

New AssemblyAI accounts get $50 in free credits to start, which covers both async and real-time transcription. Text-to-Speech is $15 per 1M characters, billed on input characters. See the AssemblyAI pricing page for current Universal-3.6 Pro Realtime streaming and Text-to-Speech rates.

Can I use a different LLM or TTS provider in this voice bot?

Yes—the architecture is modular. You can swap OpenAI for another LLM, or rewrite the generate_audio method for a different TTS provider, without changing the AssemblyAI streaming layer. The speech-to-text step is the foundation, and keeping Universal-3.6 Pro Realtime in place preserves the entity accuracy, context carryover, and low latency the rest of the pipeline depends on. Keeping text-to-speech on AssemblyAI as well means one SDK and one API key for both ends of the conversation.