Skip to main content

Overview

The circuit breaker checks how a model is performing before the LLM Gateway sends your request to it. If the model is slower or failing more than you allow, the gateway skips it. It moves to your next fallback model, or rejects the request if you have none. Fallbacks react to a request that has already failed. The circuit breaker reacts to a model that is degrading, so a slow or failing model costs you nothing extra. The circuit breaker is available on chat completions and decisions requests.

Basic usage

Set at least one threshold in fallback_config.circuit_breaker_config, and add a fallback to move to when the breaker trips:
In this example, the gateway skips gemini-2.5-flash-lite and uses claude-sonnet-4-6 when either of these is true:
  • The model’s current median latency is more than 100% above its normal latency (more than double).
  • More than 20% of its recent requests failed.

Thresholds

The circuit breaker only runs when you set at least one of these thresholds. You can set both. The breaker trips when either one is exceeded. The baseline is the normal latency of the model in the region that handles your request. The current latency and failure rate are measured by the LLM Gateway on an ongoing basis.
If the gateway has no recent measurements for a model, there is nothing to compare against, so the request is sent to the model as usual. The circuit breaker never blocks a request because of missing data.
To turn the circuit breaker off for a request, set fallback_config.circuit_breaker to false.

What happens when it trips

The circuit breaker checks each model before it is called, including each fallback.
  1. If the model is healthy, the request is sent to it.
  2. If the model trips the breaker and you set a fallback, the gateway moves to the next fallback. The response looks as if you had sent the request to the fallback model, and you are billed only for that model.
  3. If the model trips the breaker and there is no fallback left, the request fails with a 500 error.
The circuit breaker only moves to a fallback model that you listed. It doesn’t retry the same model. How many fallbacks it can walk through follows fallback_config.depth, as described in Model fallbacks.

Rejected requests

If you don’t configure any fallbacks, a tripped circuit breaker fails the request automatically. This is true even though fallback_config.retry is on by default. The automatic retry only runs after a request to a model fails, and it would send the request to the same model again, so the circuit breaker skips it. To keep the request alive when the breaker trips, add a fallback. When there is no fallback to move to, the response is a 500 with a message that names the threshold that was exceeded, and details of what was measured:
If the failure threshold tripped, the message ends with failure threshold, reason is failure_threshold, and failure_rate is returned in place of the latency fields. A rejected request is not billed. You can retry it, or catch the 500 and send the request to a different model yourself.

Choosing thresholds

  • Start loose and tighten. Latency varies from request to request, so a very low percent_threshold such as 10 can trip on normal variation. A value of 100 to 200 catches a model that has clearly slowed down.
  • Use failure_threshold for availability. It protects you from a model or provider that is returning errors, which complements fallbacks that only act after your own request has failed.
  • Always set a fallback for production traffic. Without one, a tripped breaker returns an error to your user.

API reference

Request parameters

Next steps