Skip to main content
Connect Twilio Programmable Voice to the Voice Agent API so callers can have real-time conversations with your agent over the phone. Twilio handles the phone network, your server bridges audio between Twilio Media Streams and the Voice Agent API, and the agent handles speech-to-speech. Because Twilio’s native G.711 μ-law format is byte-compatible with the Voice Agent API’s audio/pcmu encoding, the server forwards audio as-is with zero transcoding. 
Caller ↔ Twilio Media Streams ↔ Your server ↔ Voice Agent API

​
Before you begin

To complete this guide, you need:

​
Quickstart

Clone the example repo and get a working Twilio voice agent in minutes.
1

Clone the repo and install dependencies

git clone https://github.com/AssemblyAI/voice-agent-api-twilio-example.git
cd voice-agent-api-twilio-example
npm install
2

Start an ngrok tunnel

Twilio needs a public URL to reach your local server. In a separate terminal, start ngrok:
ngrok http 3000
Copy the https://...ngrok.app URL from the output.
3

Configure environment variables

cp .env.example .env
Open .env and fill in your keys:
ASSEMBLYAI_API_KEY=YOUR_API_KEY
HOSTNAME=https://your-ngrok-domain.ngrok.app
4

Run the server

npm run dev
You should see Server running on http://localhost:3000.
5

Point Twilio at your server

In the Twilio Console, open your phone number’s Voice configuration and set:
  • A call comes in → Webhook → POST → https://<your-ngrok-domain>/twiml
  • Call status changes (optional) → Webhook → POST → https://<your-ngrok-domain>/call-status
6

Call your number

Dial your Twilio number from any phone. You should hear the agent’s greeting, then have a real-time conversation. Watch the server logs to see the event stream.

​
How it works

When a call comes in, the following sequence happens:
  1. A caller dials your Twilio number.
  2. Twilio sends a webhook to POST /twiml on your server. The server returns TwiML containing a <Stream> element pointed at your WebSocket endpoint.
  3. Twilio opens a Media Streams WebSocket and starts sending the caller’s audio (G.711 μ-law, 8 kHz).
  4. Your server opens a parallel WebSocket to the Voice Agent API and sends a session.update with the system prompt, voice, greeting, tools, and audio format set to audio/pcmu.
  5. Once session.ready fires, the server forwards audio in both directions:
    • Caller → Agent: Each Twilio media event becomes an input.audio event.
    • Agent → Caller: Each reply.audio event becomes a Twilio media action.
  6. When the caller barges in (input.speech.started), the server sends a Twilio clear action so the agent stops talking immediately.

​
Return TwiML with a stream

When Twilio receives a call, it hits your /twiml endpoint. The server responds with TwiML that opens a Media Streams WebSocket: 
app.post("/twiml", (req, res) => {
  const callId = newCallId();
  const hostname = process.env.HOSTNAME.replace(/^https?:\/\//, "");
  const streamUrl = `wss://${hostname}/media-stream/${callId}`;

  res.type("text/xml").status(200).send(
    `<Response>
  <Connect>
    <Stream url="${streamUrl}" />
  </Connect>
</Response>`,
  );
});

​
Connect to the Voice Agent API

When Twilio opens the Media Streams WebSocket, the server creates a parallel connection to the Voice Agent API and sends the session configuration: 
const aaiWs = new WebSocket("wss://agents.assemblyai.com/v1/realtime", {
  headers: { Authorization: `Bearer ${process.env.ASSEMBLYAI_API_KEY}` },
});

aaiWs.send(JSON.stringify({
  type: "session.update",
  session: {
    system_prompt: "You are a helpful voice assistant.",
    greeting: "Hi, thanks for calling. How can I help?",
    input: { type: "audio", format: { encoding: "audio/pcmu" } },
    output: {
      type: "audio",
      voice: "ivy",
      format: { encoding: "audio/pcmu" },
    },
    tools: [/* your tool definitions */],
  },
}));
Both input and output use audio/pcmu (G.711 μ-law at 8 kHz) to match Twilio’s native codec. This means no transcoding or resampling is needed.

​
Bridge audio between Twilio and the Voice Agent API

Once session.ready fires, forward audio payloads in both directions: 
// Twilio → Voice Agent API
tw.on("media", (msg) => {
  if (msg.media.track !== "inbound") return;
  aaiWs.send(JSON.stringify({
    type: "input.audio",
    audio: msg.media.payload,
  }));
});

// Voice Agent API → Twilio
aaiWs.on("message", (data) => {
  const event = JSON.parse(data.toString());

  if (event.type === "reply.audio" && event.data) {
    tw.send({
      event: "media",
      streamSid: tw.streamSid,
      media: { payload: event.data },
    });
  }
});

​
Handle barge-in

When the caller starts speaking while the agent is talking, clear the Twilio audio buffer so the agent stops immediately: 
if (event.type === "input.speech.started") {
  tw.send({ event: "clear", streamSid: tw.streamSid });
}

​
Make outbound calls

The example repo also supports outbound calling. Set the Twilio credentials in .env: 
TWILIO_ACCOUNT_SID=YOUR_TWILIO_ACCOUNT_SID
TWILIO_AUTH_TOKEN=YOUR_TWILIO_AUTH_TOKEN
TWILIO_PHONE_NUMBER=+15551234567
TARGET_PHONE_NUMBER=+15557654321
With the server still running, open a new terminal and run: 
npm run outbound
This places a call from your Twilio number to the target. Twilio fetches /outbound-twiml, which connects the call to /outbound-stream. The agent speaks first using the configured greeting.

​
Add custom tools

The example includes one tool, generate_random_number. To add your own tools:
  1. Define the tool in the TOOLS array in src/bot.ts:
export const TOOLS = [
  {
    type: "function",
    name: "generate_random_number",
    description: "Generate a random integer between min and max (inclusive).",
    parameters: {
      type: "object",
      properties: {
        min: { type: "number", description: "Minimum value (inclusive)." },
        max: { type: "number", description: "Maximum value (inclusive)." },
      },
      required: ["min", "max"],
    },
  },
];
  1. Add the handler in runTool:
export async function runTool(
  name: string,
  args: Record<string, any>,
): Promise<string> {
  switch (name) {
    case "generate_random_number": {
      const min = Math.ceil(args.min);
      const max = Math.floor(args.max);
      const result = Math.floor(Math.random() * (max - min + 1)) + min;
      return JSON.stringify({ result, min: args.min, max: args.max });
    }
    default:
      return JSON.stringify({ error: `Unknown tool: ${name}` });
  }
}
  1. When the agent calls a tool, the Voice Agent API sends a tool.call event. The server runs the tool and sends back a tool.result event with the same call_id. The agent then continues the conversation naturally.
For more on tool calling, see Add tools to your agent.

​
Troubleshooting

  • Call connects but no audio: Check that HOSTNAME matches your ngrok domain and that your server is reachable. Watch ngrok’s request log for the incoming Media Streams WebSocket.
  • session.error with invalid_value on the voice field: Voice names are case-sensitive. Use lowercase (ivy, claire, dawn, etc.). See Choose a voice for available voices.
  • Greeting plays but later replies don’t: Make sure your tool handler always sends a tool.result back. The agent waits for it before continuing.
  • Audio is choppy or echoey: Twilio handles echo cancellation on the carrier side. If you hear echo during testing, it’s likely your speakerphone. Use a headset.

​
Next steps