Sequence diagram
Messages at a glance
Client → server
Server → client
Session initialization
When the session begins, you receive aBegin message with the session ID, expiration time, and the configuration the server applied to your session.
expires_at is a Unix timestamp (seconds). When it is reached, the server closes the session with error 3008 after the current turn ends, or up to 60 seconds later. You’re billed for this extra time.
Sending audio
Audio is sent as binary WebSocket frames containing audio bytes in the encoding and sample rate you set at connection time (default: 16 kHz, 16-bit mono PCM). Do not wrap audio in JSON and do not base64-encode it. A text frame that isn’t valid JSON is rejected and closes the session. Compressed audio is supported via Opus and AAC. Opus uses roughly 8× less bandwidth than 16 kHz PCM.encoding=ogg_opus— an Ogg-encapsulated Opus byte stream (the format produced by ffmpeg, gstreamer, opusenc, and browser MediaRecorder). Binary frames can be arbitrary chunks of the stream; no page or packet alignment is required.encoding=opus— raw Opus packets without a container. Each binary frame must contain exactly one Opus packet.encoding=aac— an AAC stream in ADTS framing. Raw AAC packets and MP4/M4A containers are not supported. Binary frames can be arbitrary chunks of the stream; the server buffers partial frames.
sample_rate is ignored because each stream describes its own sample rate.
For PCM, send audio in chunks of roughly 50 ms (800 samples at 16 kHz); for Opus, one 20 ms packet (or page) per frame is typical. You can send audio faster than real time (for example when streaming from a file), but throughput is throttled at ~1.25× real time. If more than five minutes of audio is buffered ahead of processing, the session closes with error 3007.
- Python
- Python SDK
- Javascript
- JavaScript SDK
Speech started
Immediately before the firstTurn message of each turn, the server sends a SpeechStarted message. The timestamp field is the start of the turn, in milliseconds relative to the beginning of the audio stream. The confidence field is the average word confidence of the initial transcript.
Partial transcript
As the speaker is talking, the server emits one or moreTurn messages with end_of_turn: false. These are partial transcripts.
End of turn
When the turn ends, the server emits aTurn message with end_of_turn: true and turn_is_formatted: true. This is the last message for this turn_order and the transcript is fully formatted.
Forcing an endpoint
To end the current turn immediately, send aForceEndpoint message:
- Python
- Python SDK
- Javascript
- JavaScript SDK
end_of_turn: true message(s) for the current turn right away, without waiting for silence or punctuation. If no turn is in progress (no words have been spoken since the last final), a final Turn message is still sent with an empty transcript.
To have your client decide when every turn ends (e.g. an external turn detection model, your own VAD, or a push-to-talk button), use manual mode on assemblyai-stt-3-7-realtime-agent.
Updating configuration mid-session
You can change transcription settings at any point afterBegin without reconnecting. Only the fields you include are changed:
- Python
- Python SDK
- Javascript
- JavaScript SDK
Keep alive
KeepAlive messages are not required. By default, sessions remain open until explicitly terminated or until the 3-hour maximum session duration is reached.
KeepAlive is only relevant if you have configured the inactivity_timeout connection parameter, which closes the session after a period of no audio or messages being sent. If you are using inactivity_timeout and want to keep the session open during periods where no audio is being sent, send a KeepAlive message to reset the inactivity timer:
- Python
- Python SDK
- Javascript
- JavaScript SDK
3006 and the message Session terminated due to inactivity: No messages received for N seconds.
Ordering guarantees.
turn_order is monotonically increasing, and all messages for a turn are delivered before the first message of the next turn. Within a turn, each Turn message supersedes the previous one. Render the latest transcript; do not append. A turn is complete on the message where both end_of_turn and turn_is_formatted are true.Heartbeat
When you setsession_heartbeat=true at connection time, or toggle it on with UpdateConfiguration, the server sends a Heartbeat message every 5 seconds of wall-clock time for the life of the session, during both speech and silence. Use it to confirm the session is alive and to watch how the client’s audio delivery compares to real time.
total_audio_received_msis the total audio received since the session started.total_duration_msis the wall-clock time since the session started.realtime_factoris the ratio of audio received to wall-clock time over the window since the previous heartbeat. It sits around 1.0 when the client streams in real time and drops below 1.0 when the client is under-delivering audio.max_speech_probabilityis the maximum voice-activity speech probability observed since the previous heartbeat. It runs near 1.0 during speech and near 0.0 during silence.
Session termination
To end a session, send aTerminate message:
- Python
- Python SDK
- Javascript
- JavaScript SDK
Terminate, keep reading from the WebSocket until you receive the Termination message. The server first flushes any in-flight messages, including the final (and formatted) Turn for audio you already sent.
Closing the socket as soon as you send Terminate silently discards your last transcript.
The Termination message is always the last message:
audio_duration_secondsis the total audio processed in the session.session_duration_secondsis the total wall-clock time the connection was open. This is the duration the session is billed on.
Termination, no further messages are sent and the server closes the connection with code 1000.
Errors
If the session fails at any point, the server sends anError message as a text frame and then closes the connection: