Voice Agent API

Browser integration

Connect browser-based apps to the Voice Agent API using a temporary token.

Connect a browser to the Voice Agent API in two steps:

  1. Your server calls GET /v1/token with your API key to mint a short-lived temporary token.
  2. Your browser opens the WebSocket with ?token=<token>, no API key exposed.

Your API key never leaves your server. Each token is single-use, it starts exactly one session, and all usage is attributed to the key that generated it.

Browsers provide built-in acoustic echo cancellation through getUserMedia, so browser-based clients work hands-free without headphones. If you’re developing on a laptop, the browser integration is the recommended starting point.

1. Generate a token on your server

Call GET /v1/token with your API key in the Authorization header. Pick an expires_in_seconds short enough to limit replay risk (60–300s is a good default) and an optional max_session_duration_seconds to cap the session length.

These two parameters control different things and are easy to confuse:

  • expires_in_seconds is the token redemption window — how long the client has to use this token to open a WebSocket. If the window elapses before the WebSocket is opened, the server returns a session.error with code unauthorized on the first frame instead of session.ready. Once a session.ready has been received, this value no longer applies.
  • max_session_duration_seconds is the session duration cap — how long the resulting voice agent session is allowed to run after the WebSocket is open.
GET
/v1/token
1curl -G https://agents.assemblyai.com/v1/token \
2     -H "Authorization: <apiKey>" \
3     -d expires_in_seconds=300
Try it
1// server/routes/voice-token.js
2import express from "express";
3
4const router = express.Router();
5
6router.get("/voice-token", async (_req, res) => {
7  const url = new URL("https://agents.assemblyai.com/v1/token");
8  url.searchParams.set("expires_in_seconds", "300");
9  url.searchParams.set("max_session_duration_seconds", "8640");
10
11  const response = await fetch(url, {
12    headers: { Authorization: `Bearer ${process.env.ASSEMBLYAI_API_KEY}` },
13  });
14
15  if (!response.ok) {
16    return res.status(response.status).send(await response.text());
17  }
18
19  const { token } = await response.json();
20  res.json({ token });
21});
22
23export default router;

expires_in_seconds must be between 1 and 600. max_session_duration_seconds must be between 60 and 10800 (defaults to 10800, the 3-hour maximum session duration).

Session end at max_session_duration_seconds

When the session reaches its server-side duration limit, the WebSocket closes. There is no separate “closing soon” warning event before this — if you need to finalize gracefully (e.g. play a wrap-up message, save state), run a client-side timer using the value you passed for max_session_duration_seconds and start your wrap-up a few seconds before it elapses.

Token expiry and failure modes

If a token is missing, expired, or invalid, the server rejects the handshake with an UNAUTHORIZED error (close code 1008). In browsers, this may surface as a close event with code 1006 and no body, you won’t receive a session.error event. Always fetch a fresh token immediately before each connection attempt.

If the WebSocket drops mid-session and you need to reconnect with session.resume, you’ll need a new token for the new WebSocket, the original token can’t be reused.

2. Connect from the browser with the token

Fetch the token from your server, then open the WebSocket with ?token=<token>. No Authorization header is needed.

1// browser/voice-agent.js
2const { token } = await fetch("/api/voice-token").then((r) => r.json());
3
4const wsUrl = new URL("wss://agents.assemblyai.com/v1/ws");
5wsUrl.searchParams.set("token", token);
6const ws = new WebSocket(wsUrl);
7
8ws.addEventListener("open", () => {
9  ws.send(
10    JSON.stringify({
11      type: "session.update",
12      session: {
13        system_prompt: "You are a helpful voice assistant.",
14        greeting: "Hi there! How can I help you today?",
15        output: { voice: "ivy" },
16      },
17    }),
18  );
19});
20
21ws.addEventListener("message", (event) => {
22  const message = JSON.parse(event.data);
23  // Handle session.ready, reply.audio, transcript.*, tool.call, etc.
24  console.log(message);
25});

Fetch a fresh token for every new WebSocket connection. Tokens are single-use, a dropped connection needs a new token to reconnect (including when using session.resume).

3. Browser quickstart

A complete working example that captures microphone audio, streams it to the Voice Agent API, and plays back the agent’s response. This requires two files, an HTML page and an AudioWorklet processor.

AudioWorklet processors must be loaded from a URL (audioContext.audioWorklet.addModule(url)), so you need at least two files. This example won’t work in a single-file environment like CodePen or JSFiddle without modifications. Use a local server (npx serve .) or a framework with static file support.

Create pcm-processor.js in the same directory as your HTML file:

1// pcm-processor.js - AudioWorklet that captures PCM16 from the mic
2class PCMProcessor extends AudioWorkletProcessor {
3  process(inputs) {
4    const input = inputs[0]?.[0];
5    if (input) {
6      // Convert Float32 [-1, 1] to Int16
7      const pcm16 = new Int16Array(input.length);
8      for (let i = 0; i < input.length; i++) {
9        pcm16[i] = Math.max(-32768, Math.min(32767, Math.round(input[i] * 32767)));
10      }
11      this.port.postMessage(pcm16.buffer, [pcm16.buffer]);
12    }
13    return true;
14  }
15}
16
17registerProcessor("pcm-processor", PCMProcessor);

Then create your HTML file:

1<!DOCTYPE html>
2<html lang="en">
3<head>
4  <meta charset="UTF-8">
5  <title>Voice Agent</title>
6</head>
7<body>
8  <button id="start">Start conversation</button>
9  <pre id="log"></pre>
10  <script>
11    const log = (msg) => { document.getElementById("log").textContent += msg + "\n"; };
12
13    document.getElementById("start").addEventListener("click", async () => {
14      // 1. Get token from your server (see step 1 above)
15      const { token } = await fetch("/api/voice-token").then((r) => r.json());
16
17      // 2. Force AudioContext to 24 kHz - avoids manual resampling on both
18      //    capture and playback in Chromium and Firefox. Safari ignores this
19      //    option (see Browser compatibility below) and runs at the hardware
20      //    rate, so production code should resample inside the worklet.
21      const audioCtx = new AudioContext({ sampleRate: 24000 });
22      await audioCtx.audioWorklet.addModule("pcm-processor.js");
23
24      // 3. Capture mic audio with echo cancellation enabled
25      const stream = await navigator.mediaDevices.getUserMedia({
26        audio: { echoCancellation: true, sampleRate: 24000 },
27      });
28      const source = audioCtx.createMediaStreamSource(stream);
29      const worklet = new AudioWorkletNode(audioCtx, "pcm-processor");
30
31      // 4. Connect WebSocket
32      const wsUrl = new URL("wss://agents.assemblyai.com/v1/ws");
33      wsUrl.searchParams.set("token", token);
34      const ws = new WebSocket(wsUrl);
35
36      let ready = false;
37      let playbackTime = audioCtx.currentTime;
38
39      // Send mic audio to the server once the session is ready
40      worklet.port.onmessage = (e) => {
41        if (ready && ws.readyState === WebSocket.OPEN) {
42          const b64 = btoa(String.fromCharCode(...new Uint8Array(e.data)));
43          ws.send(JSON.stringify({ type: "input.audio", audio: b64 }));
44        }
45      };
46      source.connect(worklet).connect(audioCtx.destination);
47
48      ws.addEventListener("open", () => {
49        ws.send(JSON.stringify({
50          type: "session.update",
51          session: {
52            system_prompt: "You are a helpful voice assistant. Keep responses concise.",
53            greeting: "Hi! How can I help you?",
54            output: { voice: "ivy" },
55          },
56        }));
57      });
58
59      ws.addEventListener("message", (event) => {
60        const msg = JSON.parse(event.data);
61
62        if (msg.type === "session.ready") {
63          ready = true;
64          log("Session ready, start speaking");
65        } else if (msg.type === "reply.audio") {
66          // Decode base64 PCM16 and schedule playback
67          const raw = atob(msg.data);
68          const pcm16 = new Int16Array(raw.length / 2);
69          for (let i = 0; i < pcm16.length; i++) {
70            pcm16[i] = raw.charCodeAt(i * 2) | (raw.charCodeAt(i * 2 + 1) << 8);
71          }
72          const float32 = new Float32Array(pcm16.length);
73          for (let i = 0; i < pcm16.length; i++) {
74            float32[i] = pcm16[i] / 32768;
75          }
76          const buffer = audioCtx.createBuffer(1, float32.length, 24000);
77          buffer.getChannelData(0).set(float32);
78          const src = audioCtx.createBufferSource();
79          src.buffer = buffer;
80          src.connect(audioCtx.destination);
81          const now = audioCtx.currentTime;
82          playbackTime = Math.max(playbackTime, now);
83          src.start(playbackTime);
84          playbackTime += buffer.duration;
85        } else if (msg.type === "reply.done" && msg.status === "interrupted") {
86          // Reset playback schedule to avoid stale audio
87          playbackTime = audioCtx.currentTime;
88        } else if (msg.type === "transcript.user") {
89          log("You: " + msg.text);
90        } else if (msg.type === "transcript.agent") {
91          log("Agent: " + msg.text);
92        } else if (msg.type === "session.error" || msg.type === "error") {
93          log("Error: " + msg.message);
94        }
95      });
96
97      ws.addEventListener("close", () => log("Connection closed"));
98    });
99  </script>
100</body>
101</html>

The key line is new AudioContext({ sampleRate: 24000 }). Most browsers default to the device sample rate (usually 48 kHz), so without this you’d need to manually resample both mic input and playback output. Forcing 24 kHz on the context avoids this entirely. Safari ignores this option and runs at the hardware rate — see Browser compatibility for a Safari-safe pipeline.

4. Browser compatibility

The quickstart above works as-is on Chromium-based browsers (Chrome, Edge, Brave, Arc) and Firefox. Safari has a known quirk that produces silently garbled audio if you don’t account for it.

BrowserAudioContext({ sampleRate }) honoredRecommended pipeline
Chrome / EdgeYesUse the quickstart as-is.
FirefoxYesUse the quickstart as-is.
Safari (desktop, iOS)No — runs at hardware rate (typically 48 kHz)Let AudioContext use its default rate and resample to/from 24 kHz inside the worklet (capture) and before playback.

Safari: resample inside the worklet

Safari ignores the sampleRate constructor option, so an AudioContext({ sampleRate: 24000 }) will silently run at 48 kHz on most Macs. Sending those samples to the Voice Agent API as if they were 24 kHz produces audio that sounds chipmunked or garbled.

Detect the actual context rate at runtime, send it into the worklet, and resample there:

1// browser/voice-agent.js — Safari-safe context
2const audioCtx = new AudioContext(); // let Safari pick its hardware rate
3await audioCtx.audioWorklet.addModule("pcm-processor.js");
4const worklet = new AudioWorkletNode(audioCtx, "pcm-processor", {
5  processorOptions: { inputSampleRate: audioCtx.sampleRate, targetSampleRate: 24000 },
6});
1// pcm-processor.js — linear resample to 24 kHz before posting PCM16
2class PCMProcessor extends AudioWorkletProcessor {
3  constructor(options) {
4    super();
5    const { inputSampleRate, targetSampleRate } = options.processorOptions;
6    this.ratio = inputSampleRate / targetSampleRate;
7  }
8  process(inputs) {
9    const input = inputs[0]?.[0];
10    if (!input) return true;
11    const outLength = Math.floor(input.length / this.ratio);
12    const pcm16 = new Int16Array(outLength);
13    for (let i = 0; i < outLength; i++) {
14      const sample = input[Math.floor(i * this.ratio)] ?? 0;
15      pcm16[i] = Math.max(-32768, Math.min(32767, Math.round(sample * 32767)));
16    }
17    this.port.postMessage(pcm16.buffer, [pcm16.buffer]);
18    return true;
19  }
20}
21registerProcessor("pcm-processor", PCMProcessor);

For playback, build the AudioBuffer at 24 kHz and let the context resample on output, or resample the decoded PCM16 to audioCtx.sampleRate before scheduling — the simplest version (createBuffer(1, length, 24000)) works on all current browsers.

Linear interpolation is good enough for speech at 24 kHz. If you want higher fidelity, use a windowed-sinc resampler such as libsamplerate compiled to WASM, or push the PCM16 through an OfflineAudioContext at the target rate.

Cross-browser checklist

  • User gesture required. All major browsers gate getUserMedia and AudioContext startup behind a user gesture (Safari is strictest). Start audio inside a click or touchstart handler and call await audioCtx.resume() before connecting nodes.
  • HTTPS or localhost. getUserMedia only works on secure origins.
  • Echo cancellation. Pass echoCancellation: true to getUserMedia so the agent’s TTS playing through the speakers doesn’t get re-captured by the mic.
  • Audio output sink. On iOS Safari, set the <audio playsinline> attribute or route through an AudioContext destination — autoplay and full-screen behavior differ from desktop.