Node.js voice agent with AssemblyAI Universal-3.5 Pro Realtime

Build a real-time voice agent in Node.js using the AssemblyAI Universal-3.5 Pro Realtime model (universal-3-5-pro) for speech-to-text — no Python required, no heavy framework dependencies.

Two modes in one repo:

1. Terminal agent (src/agent.js) — mic input via mic, plays TTS audio in your terminal
2. Browser server (src/server.js) — Node.js WebSocket server with a browser UI using getUserMedia

Why AssemblyAI Universal-3.5 Pro Realtime for Node.js?

*WER figures from Pipecat's open STT benchmark of real agent conversations.*

Universal-3.5 Pro Realtime's punctuation-based turn detection eliminates the need for a separate VAD library — the model checks for terminal punctuation after a short silence to decide when a speaker has actually finished, not just when they've gone quiet.

Heads up on model IDs: the older u3-rt-pro auto-routes to universal-3-5-pro on August 7, 2026 and stops accepting the old ID around September 25, 2026. This tutorial uses the current string.

Quick start

git clone https://github.com/kelsey-aai/voice-agent-nodejs-assemblyai
cd voice-agent-nodejs-assemblyai

npm install
cp .env.example .env
# Edit .env with your API keys

Terminal agent

npm start
# Speak into your mic — Ctrl+C to quit

Browser agent

npm run server
# Open http://localhost:3000

AssemblyAI WebSocket URL

const AAI_WS_URL =
  `wss://streaming.assemblyai.com/v3/ws` +
  `?speech_model=universal-3-5-pro` +
  `&encoding=pcm_s16le` +
  `&sample_rate=16000` +
  `&min_turn_silence=300` +   // ms of silence before the speculative end-of-turn
check
  `&max_turn_silence=1500` +  // hard ceiling: turn ends after this much silence 
regardless
  `&token=${ASSEMBLYAI_API_KEY}`;

Turn detection

AssemblyAI v3 uses three event types. Handle them like this:

ws.on("message", async (data) => {
  const msg = JSON.parse(data.toString());

  if (msg.type === "Begin") {
    console.log(`Session: ${msg.id}`);
  }

  if (msg.type === "Turn" && !msg.end_of_turn) {
    process.stdout.write(`\r${msg.transcript}`);
  }

  if (msg.type === "Turn" && msg.end_of_turn) {
    const reply = await generateResponse(msg.transcript);
    await speak(reply);
  }
});

Sending audio

Browser (getUserMedia + ScriptProcessor):

processor.onaudioprocess = (e) => {
  const float32 = e.inputBuffer.getChannelData(0);
  const int16 = new Int16Array(float32.length);
  for (let i = 0; i < float32.length; i++) {
    int16[i] = Math.max(-32768, Math.min(32767, Math.round(float32[i] * 32767)));
  }
  ws.send(int16.buffer);
};

Terminal (mic package):

const micStream = micInstance.getAudioStream();
micStream.on("data", (chunk) => {
  aaiWs.send(chunk); // raw PCM s16le bytes
});

Tuning turn detection

Universal-3.5 Pro Realtime uses punctuation-based end-of-turn detection. Steer it with a high-level mode preset, then fine-tune the two silence windows:

Note: end_of_turn_confidence_threshold does not apply to Universal-3.5 Pro Realtime — it belongs to the older universal-streaming models. Turn detection here is punctuation-based, controlled by the silence windows above.

Keyterm prompting (mid-session)

Inject domain-specific vocabulary after the session starts without restarting:

ws.send(JSON.stringify({
  type: "UpdateConfiguration",
  keyterms: ["AssemblyAI", "Universal-3.5 Pro Realtime", "your-product-name"],
}));

Conversation context (mid-session)

Universal-3.5 Pro Realtime can transcribe each user turn in the context of what your agent just said — after your agent asks a question, the model is primed for the answer, which sharpens short replies and spelled-out entities. Push your agent's last reply with the same UpdateConfiguration message after each agent turn:

ws.send(JSON.stringify({
  type: "UpdateConfiguration",
  agent_context: "Thanks! What's the email address on your account?",
}));

Requirements

• Node.js 18+
• npm install installs: ws, openai, elevenlabs, mic, dotenv
• macOS: afplay (built-in) for audio playback
• Linux: aplay or mpg123 for audio playback

Deploy to Railway, Render, or Fly.io

# Set environment variables in the platform dashboard, then:
npm run server

The browser server is stateless per-connection — each WebSocket session has its own AssemblyAI connection and conversation history.

Resources

• Universal-3.5 Pro Realtime streaming API reference
• AssemblyAI Node.js SDK
• AssemblyAI Playground

Frequently asked questions

How do I build a voice agent in Node.js?

Open a WebSocket from Node.js to AssemblyAI's streaming endpoint (wss://streaming.assemblyai.com/v3/ws?speech_model=universal-3-5-pro), stream microphone or browser audio as PCM, and act on the Turn messages the model sends back — when end_of_turn is true, pass the transcript to your LLM and send the reply to a TTS provider. You don't need a framework or a separate VAD library: Universal-3.5 Pro Realtime handles turn detection itself.

What's the best streaming speech-to-text API for a JavaScript/Node.js voice agent?

AssemblyAI's Universal-3.5 Pro Realtime is a strong fit for Node.js voice agents: it's a plain WebSocket (no SDK required), leads Pipecat's open real-agent benchmark at 6.99% WER, streams transcripts around 150 ms P50, and includes built-in turn detection, keyterm prompting, and Context Carryover. You connect with the ws package and handle Begin / Turn messages directly.

How do I connect to the AssemblyAI streaming WebSocket from Node.js?

Use the ws package to open wss://streaming.assemblyai.com/v3/ws with speech_model=universal-3-5-pro, encoding=pcm_s16le, and sample_rate=16000, authenticating with your API key. Stream 16 kHz PCM audio as binary messages, and parse incoming JSON messages by type: Begin (session start), Turn (partial and final transcripts), and end-of-turn signals via the end_of_turn field.

How does turn detection work without a VAD library in Node.js?

Universal-3.5 Pro Realtime does turn detection server-side using punctuation: after a short silence it checks for terminal punctuation (. ? !) to decide whether the speaker is finished, so you don't need to run a client-side VAD. You control the timing with min_turn_silence and max_turn_silence in the WebSocket URL, or shift the overall balance with the mode preset (min_latency / balanced / max_accuracy).

How do I update keyterms or context mid-conversation in Node.js?

Send an UpdateConfiguration message over the open WebSocket — no reconnect needed. Pass keyterms to boost domain vocabulary, or agent_context to feed the model your agent's most recent reply so it transcribes the next user turn in context. Both take effect on the following turn.

‍