Voice Agent API

Talk to a voice agent in your browser in four commands, then put it on a phone number.

Python starter

Python 3.9+, standard library only.

JavaScript starter

Node 18+, no dependencies.

Step 1: Clone

git clone https://github.com/AssemblyAI/voice-agent-starter-python
cd voice-agent-starter-python
cp .env.example .env

git clone https://github.com/AssemblyAI/voice-agent-starter-js
cd voice-agent-starter-js
cp .env.example .env

Step 2: Add your key

Paste your API key into .env:

.env

ASSEMBLYAI_API_KEY=<your-key>

Step 3: Publish the starter agent

python publish.py

npm run publish

Publishes the starter agent to your account and saves its ID for the next step.

Step 4: Talk to it

python deployment/browser/server.py

npm start

Open http://localhost:3000 in Chrome or Edge and start talking.

Put it on a phone number

Twilio passes calls to AssemblyAI over SIP, so there’s no media server or webhook to run. Buy a Twilio number, then add your credentials and a trunk domain you invent:

.env

TWILIO_ACCOUNT_SID=AC...                          # console.twilio.com, top of the page
TWILIO_AUTH_TOKEN=your_token_here                 # same place, hidden until you click it
TWILIO_PHONE_NUMBER=+1...                         # a number already in your account, E.164
TWILIO_TRUNK_DOMAIN=acme-agent.pstn.twilio.com    # a name you invent, unique across Twilio

python deployment/telephony/connect.py

npm run phone

That creates the SIP trunk on your domain, routes it to AssemblyAI, attaches your number to it, and binds your agent. Every step checks for existing state first, so it’s safe to re-run. Call the number.

Next steps

Write your own agent

Every file in agents/ is the request body for POST /v1/agents, sent unchanged. The starter agent is four fields:

agents/minimal.jsonc

{
  "name": "Minimal agent",
  "system_prompt": "You are a friendly assistant on a voice call. Keep every reply to one or two short sentences. Answer what was asked, lead with the answer, and skip the preamble. If you don't know something, say so. No exclamation marks.",
  "voice": { "voice_id": "anna" },
  "greeting": "Hey, what can I do for you?"
}

Copy the closest file to start your own (cp agents/http-tools.jsonc agents/my-agent.jsonc). Every field is commented. Publish again and both the browser tab and the phone number pick up the change on the next call.

Publish one of the other agents

Nine agents ship with each starter. Set AGENT= to pick one:

`AGENT=`	Demonstrates	Requires
`minimal`	The three required fields, and the defaults applied to the rest
`keyterms`	Biasing transcription toward names and jargon
`turn-taking`	Silence thresholds and interruption handling
`byo-llm`	Claude through the AssemblyAI gateway, or your own endpoint
`http-tools`	Tools that AssemblyAI calls on the agent’s behalf
`exa-search`	Web search during a call	`EXA_API_KEY`
`airtable-crm`	Reading a caller record and writing one back	`AIRTABLE_*`
`cal-booking`	Checking availability, then booking a slot	`CAL_*`
`dtmf`	PCI compliance: card entry on the keypad, never in the transcript, the logs, or the model	`DTMF_WEBHOOK_URL`

AGENT=cal-booking python publish.py
python deployment/browser/server.py

AGENT=cal-booking npm run publish
npm start

Give an agent your API keys

Values written as ${VAR} in an agent file are substituted at publish time from .env, or from agents/<name>.env for credentials only one agent uses. Both are gitignored, so the JSON is safe to commit.

Put the browser app online

Deploy the Python or JavaScript starter to Render. It prompts for ASSEMBLYAI_API_KEY and nothing else.Anyone with the URL can start sessions billed to that key.

1. Create

Create a reusable agent with one REST call, then update, list, and delete it.

2. Configure

Shape how it sounds and behaves: prompt, voice, greeting, audio, turn detection, keyterms, tools.

3. Deploy

Connect by agent_id over the API, from a browser, or to a phone number with Twilio.

Or jump straight to a topic:

Add tools: server-side HTTP tools and client-side function tools
Connect your own LLM: point the agent at an OpenAI-compatible model
Manage agents (REST): every endpoint, field, and validation rule
Browser integration: mint tokens and build your own client
Phone agents over SIP: the phone step above in full, including a Twilio CLI path
Events reference: every WebSocket event with full payloads
Build with AI coding tools: point Claude Code, Cursor, or v0 at these docs
Troubleshooting: symptom-to-fix table and support logging

No code at all? Talk to an agent in the Voice Agent playground.

Getting started

Create & manage agents

Agent behavior

Conversational experience

Deploy

After a session

Reference

API reference

Python starter

JavaScript starter

Step 1: Clone

Step 2: Add your key

Step 3: Publish the starter agent

Step 4: Talk to it

Put it on a phone number

Next steps

1. Create

2. Configure

3. Deploy

Python starter

JavaScript starter

​Step 1: Clone

​Step 2: Add your key

​Step 3: Publish the starter agent

​Step 4: Talk to it

​Put it on a phone number

​Next steps

1. Create

2. Configure

3. Deploy

Step 1: Clone

Step 2: Add your key

Step 3: Publish the starter agent

Step 4: Talk to it

Put it on a phone number

Next steps