Recordings over two minutes

Most dictations are short, and uploading while recording to /v1/transcribe/live gets them back fastest. That endpoint accepts up to 120 seconds of audio. /v1/transcribe accepts a finished recording of up to 20 minutes, with the same config and the same response. You don’t know how long the user will talk when they start, so start every dictation on the live endpoint, and send it to /v1/transcribe only when it runs past 120 seconds.

How the fallback works

When a live upload passes 120 seconds, the endpoint answers 413 with error_code: "audio_too_large" as soon as the limit is crossed. That can arrive while the user is still talking. The fallback is built around that one response:

Keep a copy of every audio frame you send to the live endpoint, from the first one.
Fall back only on 413 with error_code: "audio_too_large". Surface every other error as usual. A 413 with any other body is a different problem.
Keep recording after the 413. The user hasn’t stopped talking, so keep reading the microphone into the copy until they do.
Send the whole copy to /v1/transcribe once, with the same config, when the recording ends. Treat its response exactly like a live one. If it fails, surface its error, not the 413.
Stop at 20 minutes, the limit on /v1/transcribe.
On cancel, drop everything. Don’t send a fallback request for a recording the user discarded.

The fallback adds the upload time of the whole recording after the user stops, since /v1/transcribe reads the full body before it starts. That only applies to recordings over two minutes. Shorter ones get the live endpoint’s latency, as before.

Don’t stream a still-growing recording into /v1/transcribe with a chunked body to save upload time. The endpoint is for finished recordings, and a request held open for the rest of the dictation is not a supported use.

Example

Both examples upload 16 kHz mono 16-bit PCM while recording, and take your microphone loop as an iterator of audio frames that ends when the user stops.

Python
JavaScript (Node.js 18+)

import json
import os
import queue
import threading

import requests

API_KEY = os.environ["ASSEMBLYAI_API_KEY"]
BASE_URL = "https://dictation.assemblyai.com"
CONFIG = {"sample_rate": 16000, "channels": 1, "rewrite_mode": "smart"}
BOUNDARY = "----dictation-boundary"
MAX_RECORDING_BYTES = 20 * 60 * 16000 * 2  # 20 minutes of 16 kHz mono 16-bit PCM

HEAD = (
    f"--{BOUNDARY}\r\n"
    'Content-Disposition: form-data; name="config"\r\n'
    "Content-Type: application/json\r\n\r\n"
    f"{json.dumps(CONFIG)}\r\n"
    f"--{BOUNDARY}\r\n"
    'Content-Disposition: form-data; name="audio"; filename="audio"\r\n'
    "Content-Type: audio/pcm\r\n\r\n"
).encode()
TAIL = f"\r\n--{BOUNDARY}--\r\n".encode()
HEADERS = {
    "Authorization": API_KEY,
    "Content-Type": f"multipart/form-data; boundary={BOUNDARY}",
}


def is_live_cap_refusal(response):
    """True for the one response that triggers the fallback: 413 audio_too_large."""
    if response is None or response.status_code != 413:
        return False
    try:
        return response.json().get("error_code") == "audio_too_large"
    except ValueError:
        return False


def dictate(record):
    """Transcribe one dictation. `record` yields 16-bit PCM frames until key up."""
    frames = queue.Queue()
    live = {}

    def live_body():
        yield HEAD
        while (frame := frames.get()) is not None:
            yield frame
        yield TAIL

    def upload_live():
        try:
            live["response"] = requests.post(
                f"{BASE_URL}/v1/transcribe/live",
                headers=HEADERS, data=live_body(), timeout=90,
            )
        except requests.ConnectionError as error:
            # The route can answer 413 and close while frames are still being
            # written, so the client may see a dropped connection, not the 413.
            live["error"] = error

    uploader = threading.Thread(target=upload_live)
    uploader.start()

    # Keep reading the microphone until key up, whatever happens to the live
    # request, and keep a copy of every frame for the fallback.
    recording = bytearray()
    for frame in record:
        frames.put(frame)
        if len(recording) + len(frame) > MAX_RECORDING_BYTES:
            break  # stop recording at the 20-minute limit
        recording += frame
    frames.put(None)
    uploader.join()

    response = live.get("response")
    if response is not None and response.ok:
        return response.json()

    over_cap = is_live_cap_refusal(response) or (
        "error" in live and len(recording) > 120 * 16000 * 2
    )
    if not over_cap:
        if response is not None:
            response.raise_for_status()
        raise live["error"]

    # Past 120 seconds: send the whole recording, with the same config, to the
    # route for finished recordings. Its response has the same shape.
    buffered = requests.post(
        f"{BASE_URL}/v1/transcribe",
        headers=HEADERS, data=HEAD + bytes(recording) + TAIL, timeout=120,
    )
    buffered.raise_for_status()
    return buffered.json()

const API_KEY = process.env.ASSEMBLYAI_API_KEY;
const BASE_URL = "https://dictation.assemblyai.com";
const CONFIG = { sample_rate: 16000, channels: 1, rewrite_mode: "smart" };
const BOUNDARY = "----dictation-boundary";
const LIVE_CAP_BYTES = 120 * 16000 * 2; // 120 s of 16 kHz mono 16-bit PCM
const MAX_RECORDING_BYTES = 20 * 60 * 16000 * 2; // 20 minutes

const encoder = new TextEncoder();
const HEAD = encoder.encode(
  `--${BOUNDARY}\r\n` +
    'Content-Disposition: form-data; name="config"\r\n' +
    "Content-Type: application/json\r\n\r\n" +
    `${JSON.stringify(CONFIG)}\r\n` +
    `--${BOUNDARY}\r\n` +
    'Content-Disposition: form-data; name="audio"; filename="audio"\r\n' +
    "Content-Type: audio/pcm\r\n\r\n"
);
const TAIL = encoder.encode(`\r\n--${BOUNDARY}--\r\n`);
const HEADERS = {
  Authorization: API_KEY,
  "Content-Type": `multipart/form-data; boundary=${BOUNDARY}`,
};

async function isLiveCapRefusal(response) {
  if (response?.status !== 413) return false;
  const body = await response.json().catch(() => ({}));
  return body.error_code === "audio_too_large";
}

// `record` is an async iterable of 16-bit PCM chunks (Uint8Array) that ends at key up.
export async function dictate(record) {
  let push;
  const live = new ReadableStream({
    start(controller) {
      controller.enqueue(HEAD);
      push = (chunk) => {
        try {
          if (chunk) controller.enqueue(chunk);
          else (controller.enqueue(TAIL), controller.close());
        } catch {
          // The live request already ended (for example with the 413).
        }
      };
    },
  });

  const liveRequest = fetch(`${BASE_URL}/v1/transcribe/live`, {
    method: "POST",
    headers: HEADERS,
    body: live,
    duplex: "half",
  }).then(
    (response) => ({ response }),
    (error) => ({ error })
  );

  // Keep reading the microphone until key up, whatever happens to the live
  // request, and keep a copy of every chunk for the fallback.
  const chunks = [];
  let recorded = 0;
  for await (const chunk of record) {
    push(chunk);
    if (recorded + chunk.length > MAX_RECORDING_BYTES) break; // 20-minute limit
    chunks.push(chunk);
    recorded += chunk.length;
  }
  push(null);

  const { response, error } = await liveRequest;
  if (response?.ok) return response.json();

  const overCap =
    (await isLiveCapRefusal(response)) || (error && recorded > LIVE_CAP_BYTES);
  if (!overCap) throw error ?? new Error(`Dictation failed: ${response.status}`);

  // Past 120 seconds: send the whole recording, with the same config, to the
  // route for finished recordings. Its response has the same shape.
  const buffered = await fetch(`${BASE_URL}/v1/transcribe`, {
    method: "POST",
    headers: HEADERS,
    body: new Blob([HEAD, ...chunks, TAIL]),
  });
  if (!buffered.ok) throw new Error(`Dictation failed: ${buffered.status}`);
  return buffered.json();
}

The Python SDK’s transcribe_live() and open_live() stop at 120 seconds and don’t fall back, so use HTTP for recordings that can run longer. Blurt, AssemblyAI’s open-source dictation app, ships the same fallback in Swift: see AssemblyAITranscriber+LiveCapFallback.swift.

Making the fallback faster

The fallback’s wait is mostly upload time, so a smaller file comes back sooner. /v1/transcribe accepts compressed audio, such as Ogg/Opus (audio/ogg) and FLAC (audio/flac). Encoding the copy before you send it cuts the upload for long recordings. FLAC is lossless; Opus is much smaller but lossy, so check transcription quality on your own audio before switching. See Audio requirements for the accepted formats.

Achieve lowest latency: the live upload this builds on
Audio requirements: the limits on each endpoint
Error handling: every status code and what to do about it

Getting started

Features

API reference

Advanced

Recordings over two minutes

How the fallback works

Example

Making the fallback faster

​How the fallback works

​Example

​Making the fallback faster

​Related pages

How the fallback works

Example

Making the fallback faster

Related pages