This page is the canonical reference for the Voice Agent WebSocket protocol: every event the client sends, every event the server emits, and the order they appear in. For the field-level schema of each event, see the Voice Agent WebSocket API reference and the Events reference.
Sequence diagram
Events at a glance
Client → server
| Event | Purpose |
|---|
session.update | Configure the agent. First message after connect; can also be sent mid-session to update mutable fields. |
session.resume | Resume a previous session by session_id. |
input.audio | Stream a chunk of PCM16 audio to the agent (base64-encoded). |
tool.result | Return the result of a tool invocation the agent requested. |
reply.create | Ask the agent to generate a reply now, optionally with one-shot instructions. |
session.end | Cleanly end the session. |
| Event | When | Purpose |
|---|
session.ready | Once, after session.update is applied | Session established. Includes the session_id. Start streaming audio. |
session.updated | After a mid-session session.update | Acknowledges the update was applied. |
input.speech.started | Per user turn | Turn detection determined the user started speaking. |
transcript.user.delta | Repeatedly during a user turn | Partial transcript of the current user utterance. |
input.speech.stopped | Per user turn | Turn detection determined the user stopped speaking. |
transcript.user | Once per user turn | Final transcript of the user’s utterance. |
reply.started | Per agent reply | Agent has begun generating a reply. |
reply.audio | Repeatedly during a reply | A chunk of the agent’s spoken response as base64 PCM16. |
transcript.agent | Per agent reply | Text of the agent’s response, sent after all reply audio has been delivered. |
tool.call | When the agent invokes a tool | The agent wants to run a registered tool. Respond with tool.result. |
reply.done | End of each reply | Marks the end of a reply. If a tool.call preceded it, send tool.result next. |
session.ended | Once, last message | Session totals and clean shutdown. Nothing follows it. |
session.error | On failure | Session- or protocol-level error. |
Session initialization
Immediately after the WebSocket opens, send a session.update to configure the agent. You have two ways to configure:
- Stored agent. Send
{ "agent_id": "<id>" } to bind to an agent created via the Agents REST API. The stored system_prompt, greeting, tools, input, and output are applied server-side.
- Inline configuration. Omit
agent_id and send system_prompt, greeting, tools, input, and output directly.
The two modes are mutually exclusive. See Deploy your agent and Session configuration for details.
{
"type": "session.update",
"session": {
"system_prompt": "You are a concise assistant.",
"greeting": "Hi! How can I help?",
"input": { "format": { "encoding": "audio/pcm" } },
"output": { "voice": "ivy", "format": { "encoding": "audio/pcm" }, "volume": 100 }
}
}
session.ready once the session is established:
{
"type": "session.ready",
"session_id": "3207b601-2054-48df-ba77-8784dfcf9fb8"
}
Save session_id — you’ll need it to reconnect with session.resume if the WebSocket drops mid-conversation.
Sending audio
After session.ready, stream audio to the agent as input.audio events. Each event carries a base64-encoded chunk of PCM16 audio:
{
"type": "input.audio",
"audio": "<base64-encoded PCM16>"
}
session.input.format. See Audio format for the full format specification and Encoding for supported encodings.
Do not send input.audio before session.ready. Audio sent before the session is established is discarded.
User turn
When turn detection decides the user has started speaking, the server emits input.speech.started. As speech continues, partial transcripts stream back as transcript.user.delta. When the user stops speaking, the server emits input.speech.stopped followed by a final transcript.user:
{
"type": "input.speech.started"
}
{
"type": "transcript.user.delta",
"delta": "hey there"
}
{
"type": "input.speech.stopped"
}
{
"type": "transcript.user",
"transcript": "Hey there, what's the weather in Paris?"
}
Agent reply
Once the user turn ends, the agent generates a reply. Every reply is bracketed by reply.started and reply.done. In between, audio streams back as reply.audio events (base64 PCM16), and the full text arrives as transcript.agent after all audio has been delivered:
{ "type": "reply.started" }
{ "type": "reply.audio", "audio": "<base64 PCM16 chunk>" }
{
"type": "transcript.agent",
"transcript": "It's 18 degrees and sunny in Paris right now."
}
transcript.agent is emitted after the final reply.audio chunk, so it can be used to display the agent’s reply once playback completes.
tool.call event. The current reply ends with reply.done, and the client is expected to run the tool and return the result via tool.result. A fresh reply.started … reply.done cycle follows once the tool result is received:
{
"type": "tool.call",
"tool_call_id": "call_01H...",
"name": "get_weather",
"arguments": { "city": "Paris" }
}
{
"type": "tool.result",
"tool_call_id": "call_01H...",
"result": { "temperature_c": 18, "condition": "sunny" }
}
Updating configuration mid-session
You can send another session.update at any time to change mutable fields such as system_prompt, input.turn_detection, or output.volume. The server acknowledges with session.updated. Changing immutable fields (for example greeting, output.voice, or output.format) raises immutable_field. See Mutability after session.ready.
Session termination
To end a session cleanly, send session.end. The server flushes any in-flight events, emits session.ended, then closes the WebSocket with code 1000:
{ "type": "session.end" }
{
"type": "session.ended",
"reason": "client_ended"
}
Always terminate sessions explicitly. Sessions that are not ended remain open and continue to accrue charges until the server auto-closes them.
Errors
If the session fails at any point, the server sends a session.error event and closes the connection:
{
"type": "session.error",
"error_code": "immutable_field",
"message": "Field 'session.output.voice' cannot be changed after session.ready."
}
