Building a Voice AI application means combining speech-to-text with language model processing, and that second step is where complexity accumulates fast. Which model handles summarization best? What happens when a provider rate-limits you mid-call? How do you A/B test GPT-6.1 Sol against Claude Sonnet 5.5 without rewriting your integration? The growing LLM gateway market exists because these problems are universal, and when your primary model goes down at 2am, you need infrastructure that handles it automatically.
AssemblyAI's LLM gateway solves this from inside the same platform you already use for transcription. One OpenAI-compatible API endpoint. 55+ models from more than 10 providers. Zero markup on pricing. And because it runs inside the same infrastructure as your speech-to-text pipeline, there's no extra network hop eating into your latency budget.
This guide explains what AssemblyAI's LLM gateway is, which models it supports today, how it compares to alternatives, and why it was built for Voice AI rather than as a generic multi-model proxy.
What is AssemblyAI's LLM gateway?
AssemblyAI's LLM gateway is an OpenAI-compatible API that routes your transcripts through leading LLMs from OpenAI, Anthropic, Google, Moonshot, DeepSeek and others, without managing separate API keys, accounts, or billing relationships. Change one base URL, keep your existing code, and you're running on a different model.
Unlike general-purpose LLM proxies, LLM gateway is built to work with AssemblyAI transcripts. Pass a transcript ID with your request and the transcript text goes straight into your prompt, with no copy step between your speech-to-text and LLM calls.
Because LLM gateway runs inside the same infrastructure as AssemblyAI's transcription pipeline, there's no additional network hop between your speech-to-text and your LLM call. Every utterance in a voice agent flows through a single system: speech in, LLM processing, action out. That architecture difference matters when you're optimizing for sub-second response times.
Here's what changes when you add AssemblyAI's LLM gateway to your Voice AI stack:
Without LLM gateway | With LLM gateway |
Separate API keys and billing per LLM provider | One AssemblyAI API key for all 55+ models across 10+ providers |
Copying transcript text into every prompt | Pass a transcript ID; the transcript text is inserted into your prompt |
Brittle model-switching code | Swap providers with a single parameter change (OpenAI-compatible API) |
Rebuild integrations for every new model | New models added as providers release them |
5-5.5% markup on model costs (OpenRouter, llmgateway.io) | Zero markup: pay what providers charge |
Client-side retry logic when models fail or time out | Automatic cross-provider fallbacks with the same response schema |
Third-party proxy libraries with massive dependency trees | Minimal-dependency architecture built by hand |
Extra network hop between STT and LLM | Runs inside the same infrastructure as the transcription pipeline |
Access 55+ models from OpenAI, Anthropic, Google, Moonshot, DeepSeek, and more through one endpoint, with zero markup on provider costs.
When do you need AssemblyAI's LLM gateway?
LLM gateway makes sense when your application involves spoken data and you want to apply language models to it without managing the complexity yourself. Here are the scenarios where it delivers the most value:
- You're already using AssemblyAI for transcription. You don't need a new account, a new API key, or a new billing relationship. Your existing credentials work, and your dashboard shows usage across both transcription and LLM calls.
- You want to compare models without rewriting code. Testing GPT-6.1 Sol against Claude Sonnet 5.5? Change the
modelparameter fromgpt-6.1-soltoclaude-sonnet-5-5. The API contract stays identical, so your integration code doesn't need to know which model is handling the request. - You need automatic fallbacks. If your primary model errors, or its latency or failure rate crosses a threshold you set, LLM gateway can reroute to a backup model. Same schema, same response format. No client-side retry logic required.
- Pricing markup matters to you. OpenRouter charges 5-5.5% on top of provider costs. llmgateway.io charges 5%. AssemblyAI charges zero markup: you pay what the model providers charge. At scale, that difference adds up.
- You're building voice agents. LLM gateway is the LLM step between AssemblyAI-STT-3.7-Realtime-Agent for speech-to-text and AssemblyAI's Text-to-Speech API for voice output, orchestrated with LiveKit or your own loop. See the voice agent best practices. If you're building conversational voice interfaces, this keeps the whole stack on one API key and one bill.
- You're concerned about supply chain security. In March 2026, LiteLLM, a popular open-source LLM proxy library, was hit by a supply chain attack that stole API credentials. AssemblyAI was unaffected because LLM gateway doesn't depend on third-party proxy libraries. Every provider integration is built by hand, with a minimal dependency tree.
Supported models
LLM gateway provides access to 55+ models from more than 10 providers. The table below shows the current models in each family. The model catalog lists every model with its price, context window and regions.
Provider | Current models | Use cases |
OpenAI | GPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra, GPT-6 Luna, GPT-5.6 Luna, gpt-oss-120b | Complex reasoning, function calling, structured outputs; GPT-6 Luna for high-volume, low-cost work |
Anthropic | Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 5.5, Claude Sonnet 4.6, Claude Haiku 4.5 | Long-context summarization, nuanced analysis, careful instruction following |
Gemini 4 Carbon, Gemini 4 Neon, Gemini 4 Argon, Gemini 3.8 Flash, Gemini 3.7 Flash, Gemini 3.5 Flash Lite, Gemma 4 31B | 1M-token context windows, fast inference, low cost per hour of audio | |
Moonshot, DeepSeek, Z.ai, MiniMax | Kimi K3.1 Pro, Kimi K3, DeepSeek V4.1 Flash, GLM-5.3, GLM-5.3 Flash, MiniMax M3.1 Flash, MiniMax M3 | Open-weight models at lower prices, with context windows up to 1M tokens |
Nvidia, Qwen, Meta, xAI, Reflection AI, Inception | Nemotron 3 Super 120B, Qwen3 Next 80B, Muse Spark Open, Grok 4.7, Beam 1.0, Mercury Voice | Specialized and open-weight options, including Mercury Voice for voice agents |
AssemblyAI | Qwen3.5 4B Fast | Hosted by AssemblyAI for low-latency, low-cost tasks such as dictation cleanup |
This is a curated catalog of tested models, so you aren't left sorting through 300+ options to find the ones that work on speech. Models are scored on Voice Standard Bench across summarization, information extraction, text correction and dictation cleanup on real transcripts, with cost per hour of audio and response time, so you can compare models before you pick one. The newest additions, such as Gemini 4 Carbon and Gemini 4 Neon, join the results after the next benchmark run.
Note on model availability: Models are added as providers release them and retired when providers sunset them. Two current models already have retirement dates: GPT-4.1 on January 13, 2027 and Gemini 3.1 Flash Lite on May 6, 2027. Check the model catalog for the current list.
Key features
Unified interface with one API key
LLM gateway exposes an OpenAI-compatible chat completions endpoint. If you've written code against OpenAI's API, you already know the interface. Change the base URL to AssemblyAI's endpoint, swap in your AssemblyAI API key, and you're routing through LLM gateway.
The practical impact: you can switch between GPT-6.1 Sol and Claude Sonnet 5.5 by changing one parameter. No SDK changes. No authentication rework. No schema translation.
For teams already using AssemblyAI for transcription, this means one API key for your entire Voice AI stack. One set of rate limits to manage. One dashboard for monitoring. One invoice at the end of the month.
Built for transcripts
This is the core differentiator. Instead of copying transcript text between APIs, you pass a transcript_id with your request and put a {{ transcript }} tag in your prompt. LLM gateway inserts the transcript text before it runs the completion. The apply LLMs to audio files guide walks through it in Python and JavaScript.
Voice applications need different context than text applications. A meeting summary, a call score and a voice agent reply each depend on what was said, by whom, and in what order, so the transcript is the input that matters most.
LLM gateway also runs inside the same infrastructure as AssemblyAI's transcription pipeline. There's no extra network hop between your speech-to-text output and your LLM input. For voice agents where every millisecond counts, eliminating that round-trip matters. Speech flows in, gets transcribed, hits the LLM, and returns an action, all within a single system.
Full chat completion capabilities
LLM gateway supports the OpenAI chat completions feature set:
- Streaming responses: Get tokens as they're generated for real-time UIs
- Function calling: Define tools and let models invoke them with structured arguments
- JSON mode: Force structured JSON outputs for downstream processing
- System prompts: Set context and behavior across conversation turns
- Multi-turn conversations: Maintain conversation history for contextual responses
Whatever you're doing with OpenAI's API today, LLM gateway supports it, while giving you access to Claude, Gemini, Kimi, DeepSeek and more through the same interface.
Zero-markup pricing
AssemblyAI charges what model providers charge. No percentage fee on top.
To put this in perspective:
- OpenRouter: 5-5.5% markup on provider costs
- llmgateway.io: 5% markup on provider costs
- AssemblyAI LLM gateway: 0% markup
At low volume, the difference is negligible. At scale, processing thousands of transcripts per day through GPT-6.1 Sol or Claude Opus 5.5, those percentages translate to real dollars.
One detail to know: since July 1, 2026, requests to the in-region US and EU endpoints cost 10% more than global routing. That's a direct pass-through of provider price increases, with no AssemblyAI upcharge. To keep the standard rate, opt into global routing.
Billing is unified with your existing AssemblyAI account. One invoice shows transcription usage and LLM usage together. No separate payment methods. No reconciling bills from multiple vendors.
Automatic cross-provider fallbacks
Here's a scenario: your application uses GPT-6.1 Sol as the primary model. OpenAI has an outage, your request hits rate limits, or latency spikes. What happens?
With direct API calls or basic proxies, your request fails. You catch the error, implement retry logic, maybe try a different provider, handle the different response format, and hope your fallback is available.
With LLM gateway, you list fallback models in the request, up to two deep. If the primary model fails, the request reroutes to your backup: Claude Sonnet 5.5, Gemini 4 Argon, Kimi K3, whatever you've specified. A circuit breaker can also switch to the fallback when p50 latency or the failure rate rises past thresholds you set. The response comes back in the same schema, so your client code doesn't know anything changed.
For voice agents, a failed LLM call means dead air. For real-time applications, a 10-second timeout means a broken experience. Automatic fallbacks keep your application responsive when individual providers have problems.
Minimal-dependency security
In March 2026, LiteLLM, one of the most widely used open-source LLM proxy libraries, was compromised in a supply chain attack. Malicious code in a dependency update exfiltrated API keys to external servers. Teams using LiteLLM had their OpenAI, Anthropic, and other credentials exposed.
AssemblyAI was unaffected. LLM gateway doesn't use LiteLLM or similar third-party proxy libraries. Every provider integration is built by hand, with intentionally minimal dependencies, so there's no massive dependency tree where a single compromised package can steal your credentials.
This architecture decision predates the LiteLLM incident. It's a deliberate choice to control security-critical code paths, and the incident validates the approach. If you're routing production traffic and API keys through infrastructure, you want that infrastructure to have the smallest possible attack surface.
Always up-to-date model support
When a provider ships a new model, you don't rebuild your integration. Gemini 4 Carbon and Gemini 4 Neon reached LLM gateway on November 19, 2026, and Kimi K3.1 Pro and MiniMax M3.1 Flash were added earlier in November. Your code stays the same; you change the model parameter to access new capabilities.
When providers deprecate older models, AssemblyAI publishes retirement dates and migration guidance, and fallback configurations keep working. The model landscape changes constantly, and LLM gateway absorbs that churn so your application code doesn't have to.
How AssemblyAI's LLM gateway compares to alternatives
There are multiple ways to access LLMs: direct provider APIs, multi-model proxies like OpenRouter, or platform-integrated solutions like AssemblyAI's LLM gateway. Here's how they compare:
Criteria | AssemblyAI LLM gateway | OpenRouter | llmgateway.io | Direct provider APIs |
Pricing markup | 0% | 5-5.5% | 5% | 0% |
Model count | 55+ curated models | 300+ models | Varies | Provider-specific |
Automatic fallbacks | Yes, cross-provider with same schema | Limited | Limited | No, manual implementation |
Voice AI integration | Native, same infrastructure as STT | No | No | Separate integration |
Dependency security | Minimal deps, built by hand | Third-party libraries | Third-party libraries | Provider SDKs only |
Setup for existing users | Same API key, no new account | New account required | New account required | Account per provider |
Transcript input | Pass a transcript ID | No | No | Manual copy into prompts |
When to choose OpenRouter: If you need access to 300+ models including niche fine-tuned variants, and you're not building voice applications, OpenRouter's breadth makes sense. The markup is the cost of that breadth.
When to choose direct APIs: If you're only using one provider and don't need fallbacks, billing consolidation, or speech-to-text integration, direct APIs are simpler. You avoid any middleware.
When to choose AssemblyAI LLM gateway: If you're building Voice AI, already using AssemblyAI, want zero markup, need automatic fallbacks, or care about supply chain security, LLM gateway is purpose-built for your use case.
Transcribe audio and route it through GPT-6, Claude, or Gemini, all from the same API. Try it in the Playground.
Using LLM gateway with streaming speech-to-text
For real-time applications such as live captioning, voice agents and call analytics, you're often processing audio as it streams rather than waiting for a complete recording. LLM gateway works alongside AssemblyAI's Streaming Speech-to-Text API.
The architecture looks like this:
- Audio streams in via WebSocket to AssemblyAI's Streaming STT
- Partial transcripts arrive with sub-second latency as speech is recognized
- Final transcripts trigger LLM gateway calls for processing
- Results return to your application for immediate action
Because LLM gateway runs inside the same infrastructure as the streaming transcription service, there's no extra network hop. The transcript doesn't leave AssemblyAI's system to reach an LLM, so the result comes back faster than if you were making a separate API call to a third-party gateway.
This matters most for voice agents. When a user finishes speaking, you want the LLM response to begin as quickly as possible. Every 100ms of added latency feels like dead air. By eliminating the network round-trip between STT and LLM, you're shaving time off every turn in the conversation.
For teams building voice agents on AssemblyAI-STT-3.7-Realtime-Agent, LLM gateway is the natural LLM step: the final transcript goes to LLM gateway, the response goes to AssemblyAI's Text-to-Speech API, and LiveKit or your own loop runs the conversation. The voice agent best practices cover how the pieces fit, so you can focus on your application logic rather than wiring up multiple vendors.
Voice AI use cases for LLM gateway
LLM gateway works best when you're applying language models to spoken data. Here are the patterns we see most often, with models that fit each one:
Use case | How LLM gateway helps | Typical models |
Meeting summarization | Process transcripts with speaker labels to generate summaries, action items, and follow-ups | Claude Opus 5.5, GPT-6.1 Sol, Gemini 4 Argon |
Call center analytics | Analyze agent/customer conversations for sentiment, compliance, and coaching opportunities | GPT-6 Luna, Claude Haiku 5.5 |
Clinical documentation | Convert patient-provider conversations into structured clinical notes | Claude Sonnet 5.5, GPT-6.1 Sol |
Customer support agents | Power voice agents that understand queries, retrieve information, and respond naturally | GPT-6 Luna, Gemini 3.8 Flash |
Sales coaching | Analyze sales calls for talk time, objection handling, and messaging adherence | Gemini 3.8 Flash, GLM-5.3 Flash |
Content repurposing | Transform podcasts, webinars, or interviews into blog posts, social content, and newsletters | Claude Sonnet 5.5, GPT-6.1 Sol |
Appointment scheduling agents | Voice agents that handle inbound scheduling calls, check availability, and confirm bookings | GPT-6 Luna, Claude Haiku 5.5 |
Outbound reminder calls | Voice agents that call patients or customers with appointment reminders and handle rescheduling | Gemini 3.5 Flash Lite, Qwen3.5 4B Fast |
Voice agent applications
Voice agents are the fastest-growing category of LLM gateway usage. These are conversational AI applications where users speak naturally and receive spoken responses: customer support lines that don't require "press 1 for billing," or clinical intake systems that gather patient information through natural conversation.
LLM gateway is well suited here because:
- Latency is critical. Running inside the same infrastructure as STT eliminates network hops.
- Fallbacks prevent dead air. If GPT-6.1 Sol is slow or unavailable, automatic failover to Claude Sonnet 5.5 keeps the conversation flowing.
- Model choice matters per turn. Fast, low-cost models such as GPT-6 Luna or Gemini 3.8 Flash handle routine turns, and you can route harder requests to a larger model through the same API.
For teams building voice agents, LLM gateway pairs with AssemblyAI-STT-3.7-Realtime-Agent for speech-to-text and AssemblyAI's Text-to-Speech API for voice output. One API key, one bill, one set of logs across all three. Wire them together with LiveKit or your own loop, and focus on your agent's logic and conversation design.
Combine AssemblyAI-STT-3.7-Realtime-Agent, LLM gateway, and AssemblyAI's Text-to-Speech API with one API key and one bill.
Getting started
If you're already an AssemblyAI customer, you can start using LLM gateway today with your existing API key. No new signup, no separate credentials. LLM gateway is billed from your account balance from the first request, and the free credits for new accounts don't cover it.
The fastest way to see LLM gateway in action is to transcribe an audio file with AssemblyAI and then pass that transcript to an LLM through the same API key. The LLM gateway quickstart includes copy-paste examples in Python, JavaScript, and curl.
Frequently Asked Questions (FAQs)
Which models does AssemblyAI's LLM gateway support?
LLM gateway supports 55+ models from more than 10 providers. That includes OpenAI's GPT-6 family, Anthropic's Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, Google's Gemini 4 Carbon, Gemini 4 Neon, Gemini 4 Argon and Gemini 3.8 Flash, and open-weight models such as Kimi K3.1 Pro, DeepSeek V4.1 Flash, GLM-5.3 and MiniMax M3.1 Flash. The model catalog lists every model with its price, context window and regions.
What is the difference between an LLM gateway and an AI gateway?
The terms are often used interchangeably, but there's a distinction. An "AI gateway" is a broader category that might include routing to any AI model: image generation, embedding models, speech recognition, and LLMs. An "LLM gateway" specifically routes requests to large language models.
In practice, most products marketed as "AI gateways" are mainly LLM gateways with some additional model types supported. AssemblyAI's LLM gateway focuses on language models while integrating natively with AssemblyAI's speech-to-text infrastructure, giving you both capabilities through one platform.
What is the difference between an LLM gateway and an MCP gateway?
MCP (Model Context Protocol) is an open standard from Anthropic for connecting AI models to external tools and data sources. An MCP gateway manages these tool connections and context handoffs.
An LLM gateway handles model routing and API abstraction: choosing which model processes a request, managing fallbacks, and normalizing response formats. The two can work together. An LLM gateway routes your request to the right model, while MCP-compliant integrations give that model access to tools and external context.
AssemblyAI's LLM gateway focuses on the routing and abstraction layer, with direct transcript input as a first-class feature.
Can I use LLM gateway without AssemblyAI's transcription?
Yes. LLM gateway is a standalone service that works with any text input. If you're using a different transcription provider, or working with text that isn't from speech at all, you can still route your LLM calls through AssemblyAI's gateway to get unified billing, automatic fallbacks, and zero-markup pricing.
That said, the full value of LLM gateway comes from integration with AssemblyAI's transcription pipeline. Passing a transcript ID instead of copying text, and removing the network hop between STT and LLM, are specific advantages for voice applications.
How does billing work?
LLM usage is billed per input and output token at the rates model providers charge, with no markup. Requests to the in-region US and EU endpoints cost 10% more than global routing, as a pass-through of provider price increases. Charges appear on your existing AssemblyAI invoice alongside transcription usage, and LLM gateway isn't covered by the free credits for new accounts.
Usage metrics such as tokens consumed and requests by model are available in your AssemblyAI dashboard, which makes it easy to track costs and optimize model selection.
What happens when new models are released?
AssemblyAI adds new models to LLM gateway as providers release them. You'll see announcements in release notes and documentation updates. Using a new model typically requires changing only the model parameter in your API call: no SDK updates, no schema changes.
When providers deprecate models, AssemblyAI publishes retirement dates and migration guidance. Fallback configurations keep working as individual models are sunset, so your application stays resilient. You focus on what your application does with LLM outputs, and LLM gateway handles keeping up with the providers.