LLM Gateway Overview

Overview

AssemblyAI’s LLM Gateway is a unified interface that allows you to connect with multiple LLM providers including Claude, GPT, Gemini, and more. You can use the LLM Gateway to build sophisticated AI applications through a single API.

The LLM Gateway provides access to 25+ models across major AI providers with support for:

• Basic Chat Completions - Simple request/response interactions
• Streamed Responses - Stream output as it’s generated (OpenAI models)
• Multi-turn Conversations - Maintain context across multiple exchanges
• Structured Outputs - Constrain responses to a specific JSON schema
• Tool/Function Calling - Enable models to execute custom functions
• Reasoning - Control model reasoning effort for supported models (OpenAI-compatible, Gemini 3+, Anthropic)
• Agentic Workflows - Multi-step reasoning with automatic tool chaining
• Unified Interface - One API for Claude, GPT, Gemini, Qwen, Kimi, and more
• Post-processing - Automatically repair malformed JSON responses with built-in JSON repair

Available models

By quality (LMArena Score)

By latency (per 10,000 tokens)

By provider

Anthropic Claude

OpenAI GPT

Google Gemini

Alibaba Cloud Qwen

Moonshot AI Kimi

Select a model

You can specify which model to use in your request by setting the model parameter. Here are examples showing how to use Claude 4.5 Sonnet:

1
import requests
2
3
headers = {
4
  "authorization": "<YOUR_API_KEY>"
5
}
6
7
response = requests.post(
8
    "https://llm-gateway.assemblyai.com/v1/chat/completions",
9
    headers = headers,
10
    json = {
11
        "model": "claude-sonnet-4-6",
12
        "messages": [
13
            {"role": "user", "content": "What is the capital of France?"}
14
        ],
15
        "max_tokens": 1000
16
    }
17
)
18
19
result = response.json()
20
print(result["choices"][0]["message"]["content"])

Simply change the model parameter to use any of the available models listed in the Available models section above.

Logging and troubleshooting

Every LLM Gateway response includes a request_id field — a unique identifier for that request. Persist it (along with the model, the API region, and a timestamp) for every call you make, not just when something goes wrong. If you contact support@assemblyai.com about a specific request (latency spikes, unexpected output, rate-limit errors, content moderation surprises), this ID lets us locate the exact request in our logs immediately.

We recommend logging, at minimum:

• request_id from the response body
• The model parameter used
• The API region (US: llm-gateway.assemblyai.com, EU: llm-gateway.eu.assemblyai.com)
• A timestamp for when the request was sent
• The full error response body when a non-2xx status code is returned

For details on debugging specific status codes (400/401/403/429/5xx) and what information to include when filing a support request, see the Troubleshooting page.

Next steps

• Basic Chat Completions - Learn how to send simple messages and receive responses
• Multi-turn Conversations - Maintain context across multiple exchanges
• Structured Outputs - Constrain model responses to follow a specific JSON schema
• Tool Calling - Enable models to execute custom functions
• Agentic Workflows - Build multi-step reasoning applications
• Post-processing - Automatically repair malformed JSON in model responses