Connect to Twilio

Connect Twilio Programmable Voice to the Voice Agent API so callers can have real-time conversations with your agent over the phone. Twilio handles the phone network, your server bridges audio between Twilio Media Streams and the Voice Agent API, and the agent handles speech-to-speech. Because Twilio’s native G.711 μ-law format is byte-compatible with the Voice Agent API’s audio/pcmu encoding, the server forwards audio as-is with zero transcoding.

Caller ↔ Twilio Media Streams ↔ Your server ↔ Voice Agent API

Before you begin

To complete this guide, you need:

An AssemblyAI API key with Voice Agent access.
A Twilio account with a phone number. You can buy one if you don’t have one.
Node.js 20+.
ngrok for exposing your local server to the internet.

Quickstart

Clone the example repo and get a working Twilio voice agent in minutes.

Clone the repo and install dependencies

git clone https://github.com/AssemblyAI/voice-agent-api-twilio-example.git
cd voice-agent-api-twilio-example
npm install

Start an ngrok tunnel

Twilio needs a public URL to reach your local server. In a separate terminal, start ngrok:

ngrok http 3000

Copy the https://...ngrok.app URL from the output.

Configure environment variables

cp .env.example .env

Open .env and fill in your keys:

ASSEMBLYAI_API_KEY=YOUR_API_KEY
HOSTNAME=https://your-ngrok-domain.ngrok.app

Run the server

npm run dev

You should see Server running on http://localhost:3000.

Point Twilio at your server

In the Twilio Console, open your phone number’s Voice configuration and set:

A call comes in → Webhook → POST → https://<your-ngrok-domain>/twiml
Call status changes (optional) → Webhook → POST → https://<your-ngrok-domain>/call-status

Call your number

Dial your Twilio number from any phone. You should hear the agent’s greeting, then have a real-time conversation. Watch the server logs to see the event stream.

How it works

When a call comes in, the following sequence happens:

A caller dials your Twilio number.
Twilio sends a webhook to POST /twiml on your server. The server returns TwiML containing a <Stream> element pointed at your WebSocket endpoint.
Twilio opens a Media Streams WebSocket and starts sending the caller’s audio (G.711 μ-law, 8 kHz).
Your server opens a parallel WebSocket to the Voice Agent API and binds to your stored agent by agent_id, with the audio format set to audio/pcmu.
Once session.ready fires, the server forwards audio in both directions:
- Caller → Agent: Each Twilio media event becomes an input.audio event.
- Agent → Caller: Each reply.audio event becomes a Twilio media action.
When the caller barges in (input.speech.started), the server sends a Twilio clear action so the agent stops talking immediately.

Return TwiML with a stream

When Twilio receives a call, it hits your /twiml endpoint. The server responds with TwiML that opens a Media Streams WebSocket:

app.post("/twiml", (req, res) => {
  const callId = newCallId();
  const hostname = process.env.HOSTNAME.replace(/^https?:\/\//, "");
  const streamUrl = `wss://${hostname}/media-stream/${callId}`;

  res.type("text/xml").status(200).send(
    `<Response>
  <Connect>
    <Stream url="${streamUrl}" />
  </Connect>
</Response>`,
  );
});

Connect to the Voice Agent API

When Twilio opens the Media Streams WebSocket, the server creates a parallel connection to the Voice Agent API and binds to your stored agent by agent_id:

const aaiWs = new WebSocket("wss://agents.assemblyai.com/v1/ws", {
  headers: { Authorization: process.env.ASSEMBLYAI_API_KEY },
});

aaiWs.on("open", () => {
  aaiWs.send(JSON.stringify({
    type: "session.update",
    session: { agent_id: process.env.AGENT_ID },
  }));
});

Create the agent with audio/pcmu encoding. Twilio streams G.711 μ-law at 8 kHz, so store that format on the agent — set both its input and output format.encoding to audio/pcmu when you create it. Binding by agent_id is mutually exclusive with inline input/output, so the format lives on the agent, not in this session.update. Matching Twilio’s native codec means zero transcoding or resampling.

Bridge audio between Twilio and the Voice Agent API

Once session.ready fires, forward audio payloads in both directions:

// Twilio → Voice Agent API
tw.on("media", (msg) => {
  if (msg.media.track !== "inbound") return;
  aaiWs.send(JSON.stringify({
    type: "input.audio",
    audio: msg.media.payload,
  }));
});

// Voice Agent API → Twilio
aaiWs.on("message", (data) => {
  const event = JSON.parse(data.toString());

  if (event.type === "reply.audio" && event.data) {
    tw.send({
      event: "media",
      streamSid: tw.streamSid,
      media: { payload: event.data },
    });
  }
});

Handle barge-in

When the caller starts speaking while the agent is talking, clear the Twilio audio buffer so the agent stops immediately:

if (event.type === "input.speech.started") {
  tw.send({ event: "clear", streamSid: tw.streamSid });
}

Make outbound calls

The example repo also supports outbound calling. Set the Twilio credentials in .env:

TWILIO_ACCOUNT_SID=YOUR_TWILIO_ACCOUNT_SID
TWILIO_AUTH_TOKEN=YOUR_TWILIO_AUTH_TOKEN
TWILIO_PHONE_NUMBER=+15551234567
TARGET_PHONE_NUMBER=+15557654321

With the server still running, open a new terminal and run:

npm run outbound

This places a call from your Twilio number to the target. Twilio fetches /outbound-twiml, which connects the call to /outbound-stream. The agent speaks first using the configured greeting.

Add custom tools

The example includes one tool, generate_random_number. To add your own tools:

Define the tool in the TOOLS array in src/bot.ts:

export const TOOLS = [
  {
    type: "function",
    name: "generate_random_number",
    description: "Generate a random integer between min and max (inclusive).",
    parameters: {
      type: "object",
      properties: {
        min: { type: "number", description: "Minimum value (inclusive)." },
        max: { type: "number", description: "Maximum value (inclusive)." },
      },
      required: ["min", "max"],
    },
  },
];

Add the handler in runTool:

export async function runTool(
  name: string,
  args: Record<string, any>,
): Promise<string> {
  switch (name) {
    case "generate_random_number": {
      const min = Math.ceil(args.min);
      const max = Math.floor(args.max);
      const result = Math.floor(Math.random() * (max - min + 1)) + min;
      return JSON.stringify({ result, min: args.min, max: args.max });
    }
    default:
      return JSON.stringify({ error: `Unknown tool: ${name}` });
  }
}

When the agent calls a tool, the Voice Agent API sends a tool.call event. The server runs the tool and sends back a tool.result event with the same call_id. The agent then continues the conversation naturally.

For more on tool calling, see Add tools to your agent.

Troubleshooting

Call connects but no audio: Check that HOSTNAME matches your ngrok domain and that your server is reachable. Watch ngrok’s request log for the incoming Media Streams WebSocket.
session.error with invalid_value on the voice field: Voice names are case-sensitive. Use lowercase (ivy, claire, dawn, etc.). See Choose a voice for available voices.
Greeting plays but later replies don’t: Make sure your tool handler always sends a tool.result back. The agent waits for it before continuing.
Audio is choppy or echoey: Twilio handles echo cancellation on the carrier side. If you hear echo during testing, it’s likely your speakerphone. Use a headset.

Next steps

Configure your agent: Customize the system prompt, greeting, and turn detection.
Choose a voice: Pick a voice for your agent.
Add tools to your agent: Give your agent the ability to call functions.
Audio format: Learn more about supported encodings.
Twilio Media Streams: Twilio’s documentation on Media Streams.

Getting started

Create & manage agents

Agent behavior

Conversational experience

Deploy

After a session

Reference

API reference

Before you begin

Quickstart

How it works

Return TwiML with a stream

Connect to the Voice Agent API

Bridge audio between Twilio and the Voice Agent API

Handle barge-in

Make outbound calls

Add custom tools

Troubleshooting

Next steps

​Before you begin

​Quickstart

​How it works

​Return TwiML with a stream

​Connect to the Voice Agent API

​Bridge audio between Twilio and the Voice Agent API

​Handle barge-in

​Make outbound calls

​Add custom tools

​Troubleshooting

​Next steps

Before you begin

Quickstart

How it works

Return TwiML with a stream

Connect to the Voice Agent API

Bridge audio between Twilio and the Voice Agent API

Handle barge-in

Make outbound calls

Add custom tools

Troubleshooting

Next steps