Now live New models for transcription and text-to-speech. AssemblyAI-STT-3.7 AssemblyAI-TTS-1
Insights & Use Cases

What is LLM Gateway?

LLM gateway explained: compare features like routing, failover, security, and cost control so you can choose the right option for production AI apps today.

Abstract green sphere illustration

Written by

Kelsey Foster

Published on

26 October 2026

Building a Voice AI application means combining speech-to-text with language model processing, and that second step is where complexity accumulates fast. Which model handles summarization best? What happens when a provider rate-limits you mid-call? How do you A/B test GPT-6.1 Sol against Claude Sonnet 5.5 without rewriting your integration? The growing LLM gateway market exists because these problems are universal, and when your primary model goes down at 2am, you need infrastructure that handles it automatically.

AssemblyAI's LLM gateway solves this from inside the same platform you already use for transcription. One OpenAI-compatible API endpoint. 55+ models from more than 10 providers. Zero markup on pricing. And because it runs inside the same infrastructure as your speech-to-text pipeline, there's no extra network hop eating into your latency budget.

This guide explains what AssemblyAI's LLM gateway is, which models it supports today, how it compares to alternatives, and why it was built for Voice AI rather than as a generic multi-model proxy.

What is AssemblyAI's LLM gateway?

AssemblyAI's LLM gateway is an OpenAI-compatible API that routes your transcripts through leading LLMs from OpenAI, Anthropic, Google, Moonshot, DeepSeek and others, without managing separate API keys, accounts, or billing relationships. Change one base URL, keep your existing code, and you're running on a different model.

Unlike general-purpose LLM proxies, LLM gateway is built to work with AssemblyAI transcripts. Pass a transcript ID with your request and the transcript text goes straight into your prompt, with no copy step between your speech-to-text and LLM calls.

Because LLM gateway runs inside the same infrastructure as AssemblyAI's transcription pipeline, there's no additional network hop between your speech-to-text and your LLM call. Every utterance in a voice agent flows through a single system: speech in, LLM processing, action out. That architecture difference matters when you're optimizing for sub-second response times.

Here's what changes when you add AssemblyAI's LLM gateway to your Voice AI stack:

Without LLM gateway

With LLM gateway

Separate API keys and billing per LLM provider

One AssemblyAI API key for all 55+ models across 10+ providers

Copying transcript text into every prompt

Pass a transcript ID; the transcript text is inserted into your prompt

Brittle model-switching code

Swap providers with a single parameter change (OpenAI-compatible API)

Rebuild integrations for every new model

New models added as providers release them

5-5.5% markup on model costs (OpenRouter, llmgateway.io)

Zero markup: pay what providers charge

Client-side retry logic when models fail or time out

Automatic cross-provider fallbacks with the same response schema

Third-party proxy libraries with massive dependency trees

Minimal-dependency architecture built by hand

Extra network hop between STT and LLM

Runs inside the same infrastructure as the transcription pipeline

Try LLM gateway with your existing API key

Access 55+ models from OpenAI, Anthropic, Google, Moonshot, DeepSeek, and more through one endpoint, with zero markup on provider costs.

Get your API key

When do you need AssemblyAI's LLM gateway?

LLM gateway makes sense when your application involves spoken data and you want to apply language models to it without managing the complexity yourself. Here are the scenarios where it delivers the most value:

  • You're already using AssemblyAI for transcription. You don't need a new account, a new API key, or a new billing relationship. Your existing credentials work, and your dashboard shows usage across both transcription and LLM calls.
  • You want to compare models without rewriting code. Testing GPT-6.1 Sol against Claude Sonnet 5.5? Change the model parameter from gpt-6.1-sol to claude-sonnet-5-5. The API contract stays identical, so your integration code doesn't need to know which model is handling the request.
  • You need automatic fallbacks. If your primary model errors, or its latency or failure rate crosses a threshold you set, LLM gateway can reroute to a backup model. Same schema, same response format. No client-side retry logic required.
  • Pricing markup matters to you. OpenRouter charges 5-5.5% on top of provider costs. llmgateway.io charges 5%. AssemblyAI charges zero markup: you pay what the model providers charge. At scale, that difference adds up.
  • You're building voice agents. LLM gateway is the LLM step between AssemblyAI-STT-3.7-Realtime-Agent for speech-to-text and AssemblyAI's Text-to-Speech API for voice output, orchestrated with LiveKit or your own loop. See the voice agent best practices. If you're building conversational voice interfaces, this keeps the whole stack on one API key and one bill.
  • You're concerned about supply chain security. In March 2026, LiteLLM, a popular open-source LLM proxy library, was hit by a supply chain attack that stole API credentials. AssemblyAI was unaffected because LLM gateway doesn't depend on third-party proxy libraries. Every provider integration is built by hand, with a minimal dependency tree.

Supported models

LLM gateway provides access to 55+ models from more than 10 providers. The table below shows the current models in each family. The model catalog lists every model with its price, context window and regions.

Provider

Current models

Use cases

OpenAI

GPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra, GPT-6 Luna, GPT-5.6 Luna, gpt-oss-120b

Complex reasoning, function calling, structured outputs; GPT-6 Luna for high-volume, low-cost work

Anthropic

Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 5.5, Claude Sonnet 4.6, Claude Haiku 4.5

Long-context summarization, nuanced analysis, careful instruction following

Google

Gemini 4 Carbon, Gemini 4 Neon, Gemini 4 Argon, Gemini 3.8 Flash, Gemini 3.7 Flash, Gemini 3.5 Flash Lite, Gemma 4 31B

1M-token context windows, fast inference, low cost per hour of audio

Moonshot, DeepSeek, Z.ai, MiniMax

Kimi K3.1 Pro, Kimi K3, DeepSeek V4.1 Flash, GLM-5.3, GLM-5.3 Flash, MiniMax M3.1 Flash, MiniMax M3

Open-weight models at lower prices, with context windows up to 1M tokens

Nvidia, Qwen, Meta, xAI, Reflection AI, Inception

Nemotron 3 Super 120B, Qwen3 Next 80B, Muse Spark Open, Grok 4.7, Beam 1.0, Mercury Voice

Specialized and open-weight options, including Mercury Voice for voice agents

AssemblyAI

Qwen3.5 4B Fast

Hosted by AssemblyAI for low-latency, low-cost tasks such as dictation cleanup

This is a curated catalog of tested models, so you aren't left sorting through 300+ options to find the ones that work on speech. Models are scored on Voice Standard Bench across summarization, information extraction, text correction and dictation cleanup on real transcripts, with cost per hour of audio and response time, so you can compare models before you pick one. The newest additions, such as Gemini 4 Carbon and Gemini 4 Neon, join the results after the next benchmark run.

Note on model availability: Models are added as providers release them and retired when providers sunset them. Two current models already have retirement dates: GPT-4.1 on January 13, 2027 and Gemini 3.1 Flash Lite on May 6, 2027. Check the model catalog for the current list.

Key features

Unified interface with one API key

LLM gateway exposes an OpenAI-compatible chat completions endpoint. If you've written code against OpenAI's API, you already know the interface. Change the base URL to AssemblyAI's endpoint, swap in your AssemblyAI API key, and you're routing through LLM gateway.

The practical impact: you can switch between GPT-6.1 Sol and Claude Sonnet 5.5 by changing one parameter. No SDK changes. No authentication rework. No schema translation.

For teams already using AssemblyAI for transcription, this means one API key for your entire Voice AI stack. One set of rate limits to manage. One dashboard for monitoring. One invoice at the end of the month.

Built for transcripts

This is the core differentiator. Instead of copying transcript text between APIs, you pass a transcript_id with your request and put a {{ transcript }} tag in your prompt. LLM gateway inserts the transcript text before it runs the completion. The apply LLMs to audio files guide walks through it in Python and JavaScript.

Voice applications need different context than text applications. A meeting summary, a call score and a voice agent reply each depend on what was said, by whom, and in what order, so the transcript is the input that matters most.

LLM gateway also runs inside the same infrastructure as AssemblyAI's transcription pipeline. There's no extra network hop between your speech-to-text output and your LLM input. For voice agents where every millisecond counts, eliminating that round-trip matters. Speech flows in, gets transcribed, hits the LLM, and returns an action, all within a single system.

Full chat completion capabilities

LLM gateway supports the OpenAI chat completions feature set:

  • Streaming responses: Get tokens as they're generated for real-time UIs
  • Function calling: Define tools and let models invoke them with structured arguments
  • JSON mode: Force structured JSON outputs for downstream processing
  • System prompts: Set context and behavior across conversation turns
  • Multi-turn conversations: Maintain conversation history for contextual responses

Whatever you're doing with OpenAI's API today, LLM gateway supports it, while giving you access to Claude, Gemini, Kimi, DeepSeek and more through the same interface.

Zero-markup pricing

AssemblyAI charges what model providers charge. No percentage fee on top.

To put this in perspective:

  • OpenRouter: 5-5.5% markup on provider costs
  • llmgateway.io: 5% markup on provider costs
  • AssemblyAI LLM gateway: 0% markup

At low volume, the difference is negligible. At scale, processing thousands of transcripts per day through GPT-6.1 Sol or Claude Opus 5.5, those percentages translate to real dollars.

One detail to know: since July 1, 2026, requests to the in-region US and EU endpoints cost 10% more than global routing. That's a direct pass-through of provider price increases, with no AssemblyAI upcharge. To keep the standard rate, opt into global routing.

Billing is unified with your existing AssemblyAI account. One invoice shows transcription usage and LLM usage together. No separate payment methods. No reconciling bills from multiple vendors.

Automatic cross-provider fallbacks

Here's a scenario: your application uses GPT-6.1 Sol as the primary model. OpenAI has an outage, your request hits rate limits, or latency spikes. What happens?

With direct API calls or basic proxies, your request fails. You catch the error, implement retry logic, maybe try a different provider, handle the different response format, and hope your fallback is available.

With LLM gateway, you list fallback models in the request, up to two deep. If the primary model fails, the request reroutes to your backup: Claude Sonnet 5.5, Gemini 4 Argon, Kimi K3, whatever you've specified. A circuit breaker can also switch to the fallback when p50 latency or the failure rate rises past thresholds you set. The response comes back in the same schema, so your client code doesn't know anything changed.

For voice agents, a failed LLM call means dead air. For real-time applications, a 10-second timeout means a broken experience. Automatic fallbacks keep your application responsive when individual providers have problems.

Minimal-dependency security

In March 2026, LiteLLM, one of the most widely used open-source LLM proxy libraries, was compromised in a supply chain attack. Malicious code in a dependency update exfiltrated API keys to external servers. Teams using LiteLLM had their OpenAI, Anthropic, and other credentials exposed.

AssemblyAI was unaffected. LLM gateway doesn't use LiteLLM or similar third-party proxy libraries. Every provider integration is built by hand, with intentionally minimal dependencies, so there's no massive dependency tree where a single compromised package can steal your credentials.

This architecture decision predates the LiteLLM incident. It's a deliberate choice to control security-critical code paths, and the incident validates the approach. If you're routing production traffic and API keys through infrastructure, you want that infrastructure to have the smallest possible attack surface.

Always up-to-date model support

When a provider ships a new model, you don't rebuild your integration. Gemini 4 Carbon and Gemini 4 Neon reached LLM gateway on November 19, 2026, and Kimi K3.1 Pro and MiniMax M3.1 Flash were added earlier in November. Your code stays the same; you change the model parameter to access new capabilities.

When providers deprecate older models, AssemblyAI publishes retirement dates and migration guidance, and fallback configurations keep working. The model landscape changes constantly, and LLM gateway absorbs that churn so your application code doesn't have to.

How AssemblyAI's LLM gateway compares to alternatives

There are multiple ways to access LLMs: direct provider APIs, multi-model proxies like OpenRouter, or platform-integrated solutions like AssemblyAI's LLM gateway. Here's how they compare:

Criteria

AssemblyAI LLM gateway

OpenRouter

llmgateway.io

Direct provider APIs

Pricing markup

0%

5-5.5%

5%

0%

Model count

55+ curated models

300+ models

Varies

Provider-specific

Automatic fallbacks

Yes, cross-provider with same schema

Limited

Limited

No, manual implementation

Voice AI integration

Native, same infrastructure as STT

No

No

Separate integration

Dependency security

Minimal deps, built by hand

Third-party libraries

Third-party libraries

Provider SDKs only

Setup for existing users

Same API key, no new account

New account required

New account required

Account per provider

Transcript input

Pass a transcript ID

No

No

Manual copy into prompts

When to choose OpenRouter: If you need access to 300+ models including niche fine-tuned variants, and you're not building voice applications, OpenRouter's breadth makes sense. The markup is the cost of that breadth.

When to choose direct APIs: If you're only using one provider and don't need fallbacks, billing consolidation, or speech-to-text integration, direct APIs are simpler. You avoid any middleware.

When to choose AssemblyAI LLM gateway: If you're building Voice AI, already using AssemblyAI, want zero markup, need automatic fallbacks, or care about supply chain security, LLM gateway is purpose-built for your use case.

See LLM gateway in action

Transcribe audio and route it through GPT-6, Claude, or Gemini, all from the same API. Try it in the Playground.

Open playground

Using LLM gateway with streaming speech-to-text

For real-time applications such as live captioning, voice agents and call analytics, you're often processing audio as it streams rather than waiting for a complete recording. LLM gateway works alongside AssemblyAI's Streaming Speech-to-Text API.

The architecture looks like this:

  1. Audio streams in via WebSocket to AssemblyAI's Streaming STT
  2. Partial transcripts arrive with sub-second latency as speech is recognized
  3. Final transcripts trigger LLM gateway calls for processing
  4. Results return to your application for immediate action

Because LLM gateway runs inside the same infrastructure as the streaming transcription service, there's no extra network hop. The transcript doesn't leave AssemblyAI's system to reach an LLM, so the result comes back faster than if you were making a separate API call to a third-party gateway.

This matters most for voice agents. When a user finishes speaking, you want the LLM response to begin as quickly as possible. Every 100ms of added latency feels like dead air. By eliminating the network round-trip between STT and LLM, you're shaving time off every turn in the conversation.

For teams building voice agents on AssemblyAI-STT-3.7-Realtime-Agent, LLM gateway is the natural LLM step: the final transcript goes to LLM gateway, the response goes to AssemblyAI's Text-to-Speech API, and LiveKit or your own loop runs the conversation. The voice agent best practices cover how the pieces fit, so you can focus on your application logic rather than wiring up multiple vendors.

Voice AI use cases for LLM gateway

LLM gateway works best when you're applying language models to spoken data. Here are the patterns we see most often, with models that fit each one:

Use case

How LLM gateway helps

Typical models

Meeting summarization

Process transcripts with speaker labels to generate summaries, action items, and follow-ups

Claude Opus 5.5, GPT-6.1 Sol, Gemini 4 Argon

Call center analytics

Analyze agent/customer conversations for sentiment, compliance, and coaching opportunities

GPT-6 Luna, Claude Haiku 5.5

Clinical documentation

Convert patient-provider conversations into structured clinical notes

Claude Sonnet 5.5, GPT-6.1 Sol

Customer support agents

Power voice agents that understand queries, retrieve information, and respond naturally

GPT-6 Luna, Gemini 3.8 Flash

Sales coaching

Analyze sales calls for talk time, objection handling, and messaging adherence

Gemini 3.8 Flash, GLM-5.3 Flash

Content repurposing

Transform podcasts, webinars, or interviews into blog posts, social content, and newsletters

Claude Sonnet 5.5, GPT-6.1 Sol

Appointment scheduling agents

Voice agents that handle inbound scheduling calls, check availability, and confirm bookings

GPT-6 Luna, Claude Haiku 5.5

Outbound reminder calls

Voice agents that call patients or customers with appointment reminders and handle rescheduling

Gemini 3.5 Flash Lite, Qwen3.5 4B Fast

Voice agent applications

Voice agents are the fastest-growing category of LLM gateway usage. These are conversational AI applications where users speak naturally and receive spoken responses: customer support lines that don't require "press 1 for billing," or clinical intake systems that gather patient information through natural conversation.

LLM gateway is well suited here because:

  • Latency is critical. Running inside the same infrastructure as STT eliminates network hops.
  • Fallbacks prevent dead air. If GPT-6.1 Sol is slow or unavailable, automatic failover to Claude Sonnet 5.5 keeps the conversation flowing.
  • Model choice matters per turn. Fast, low-cost models such as GPT-6 Luna or Gemini 3.8 Flash handle routine turns, and you can route harder requests to a larger model through the same API.

For teams building voice agents, LLM gateway pairs with AssemblyAI-STT-3.7-Realtime-Agent for speech-to-text and AssemblyAI's Text-to-Speech API for voice output. One API key, one bill, one set of logs across all three. Wire them together with LiveKit or your own loop, and focus on your agent's logic and conversation design.

Build voice agents on one platform

Combine AssemblyAI-STT-3.7-Realtime-Agent, LLM gateway, and AssemblyAI's Text-to-Speech API with one API key and one bill.

Sign up free

Getting started

If you're already an AssemblyAI customer, you can start using LLM gateway today with your existing API key. No new signup, no separate credentials. LLM gateway is billed from your account balance from the first request, and the free credits for new accounts don't cover it.

The fastest way to see LLM gateway in action is to transcribe an audio file with AssemblyAI and then pass that transcript to an LLM through the same API key. The LLM gateway quickstart includes copy-paste examples in Python, JavaScript, and curl.

Frequently Asked Questions (FAQs)

Which models does AssemblyAI's LLM gateway support?

LLM gateway supports 55+ models from more than 10 providers. That includes OpenAI's GPT-6 family, Anthropic's Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, Google's Gemini 4 Carbon, Gemini 4 Neon, Gemini 4 Argon and Gemini 3.8 Flash, and open-weight models such as Kimi K3.1 Pro, DeepSeek V4.1 Flash, GLM-5.3 and MiniMax M3.1 Flash. The model catalog lists every model with its price, context window and regions.

What is the difference between an LLM gateway and an AI gateway?

The terms are often used interchangeably, but there's a distinction. An "AI gateway" is a broader category that might include routing to any AI model: image generation, embedding models, speech recognition, and LLMs. An "LLM gateway" specifically routes requests to large language models.

In practice, most products marketed as "AI gateways" are mainly LLM gateways with some additional model types supported. AssemblyAI's LLM gateway focuses on language models while integrating natively with AssemblyAI's speech-to-text infrastructure, giving you both capabilities through one platform.

What is the difference between an LLM gateway and an MCP gateway?

MCP (Model Context Protocol) is an open standard from Anthropic for connecting AI models to external tools and data sources. An MCP gateway manages these tool connections and context handoffs.

An LLM gateway handles model routing and API abstraction: choosing which model processes a request, managing fallbacks, and normalizing response formats. The two can work together. An LLM gateway routes your request to the right model, while MCP-compliant integrations give that model access to tools and external context.

AssemblyAI's LLM gateway focuses on the routing and abstraction layer, with direct transcript input as a first-class feature.

Can I use LLM gateway without AssemblyAI's transcription?

Yes. LLM gateway is a standalone service that works with any text input. If you're using a different transcription provider, or working with text that isn't from speech at all, you can still route your LLM calls through AssemblyAI's gateway to get unified billing, automatic fallbacks, and zero-markup pricing.

That said, the full value of LLM gateway comes from integration with AssemblyAI's transcription pipeline. Passing a transcript ID instead of copying text, and removing the network hop between STT and LLM, are specific advantages for voice applications.

How does billing work?

LLM usage is billed per input and output token at the rates model providers charge, with no markup. Requests to the in-region US and EU endpoints cost 10% more than global routing, as a pass-through of provider price increases. Charges appear on your existing AssemblyAI invoice alongside transcription usage, and LLM gateway isn't covered by the free credits for new accounts.

Usage metrics such as tokens consumed and requests by model are available in your AssemblyAI dashboard, which makes it easy to track costs and optimize model selection.

What happens when new models are released?

AssemblyAI adds new models to LLM gateway as providers release them. You'll see announcements in release notes and documentation updates. Using a new model typically requires changing only the model parameter in your API call: no SDK updates, no schema changes.

When providers deprecate models, AssemblyAI publishes retirement dates and migration guidance. Fallback configurations keep working as individual models are sunset, so your application stays resilient. You focus on what your application does with LLM outputs, and LLM gateway handles keeping up with the providers.