LLM API Fallback: Which Failures Should Reroute a Request
The easy assumption is that any failed LLM call should be rerouted to another provider. Only four detectable failures qualify for a reroute: network connectivity issues, provider API errors on HTTP 500, 502, 503, or 504, request timeouts, and authentication failures. Rate limiting sits in between, with a bounded retry and backoff first and a reroute only if the limit persists. A 4xx for a malformed body fails the same way on the backup, so rerouting it relocates the error. What follows is the contract: triggers, constraints, and where the chain stops.
What is LLM API fallback and how is it different from a retry?
Fallback and retry are two different actions, and keeping them apart is the first decision in the contract. Fallback automatically reroutes a request to a backup model or provider when the primary fails, times out, or is rate-limited, so the caller gets a response instead of an error. A retry re-sends the same request to the same target, which suits a transient blip; a hard failure will not clear that way.
Target, condition, and risk are where the two separate.
| Decision | Fallback | Retry |
|---|---|---|
| Target | Backup model or provider | Same model or provider |
| Condition | Primary fails, times out, or is rate-limited | Transient blips, such as a momentary connection reset |
| Risk | Response quality and cost may shift because the backup differs | The same failure can repeat, so retries stay bounded |
The order matters as well. The TrueFoundry guide on LLM fallback puts bounded retries, exponential backoff, and circuit breakers ahead of any reroute. The WSO2 fallback guide describes the same sequence: retry briefly with backoff, trip a circuit breaker if the failures persist, and fall back to another provider while the breaker is open. Both describe the transient case. Fallback is the step you reach once the retry path has not succeeded.
Which failures should trigger a fallback?
Four trigger classes qualify a primary failure for rerouting: network connectivity issues, provider API errors on HTTP 500, 502, 503, or 504, request timeouts, and authentication failures. A persistent rate limit also sits in the fallback column, after a bounded retry with backoff has failed to clear it. A 4xx that reports a bad request stays out of both columns.
Each class qualifies because the gateway can see it happen, which is what makes eligibility testable. Provider API errors show up in the HTTP status. Connectivity issues show up in a connectivity error. Timeouts show up in request timing. Authentication failures show up in an authentication error. The Bifrost Mocker walkthrough enumerates these four as examples, and the test for adding a class of your own is whether it lands on a signal the gateway can observe.
Here is the contract as a table: trigger class, observable signal, reroute decision.
| Trigger class | Observable signal | Reroute decision |
|---|---|---|
| Network connectivity issues | Connectivity error reported at the gateway | Reroute permitted |
| Provider API errors | HTTP 500, 502, 503, or 504 from the provider | Reroute permitted |
| Request timeouts | Elapsed time crosses the configured timeout | Reroute permitted |
| Authentication failures | Authentication error returned by the provider | Reroute permitted |
| Rate limiting | HTTP 429 from the provider | Retry with backoff first; reroute if the limit persists |
The 4xx line is the one worth drawing carefully. A 4xx for a malformed body or an unsupported parameter fails the same way on the backup, so a reroute moves the error without fixing it. A 4xx that reports capacity or credential state behaves differently, which is why the table permits a reroute for both rate limiting and authentication errors, and not for rate limiting alone. These classes come from the Bifrost Mocker, a third-party SDK demo, rather than from MixRoute telemetry or a customer case study.
Compatibility scope is the second boundary in the eligibility contract. A reroute only does something when the backup model or provider can accept the request, so the contract has to define compatibility in terms of request format and model capability. The published material stops short of a verification method, which leaves that rule to you, in the same position as user disclosure and stop conditions.
What constraints bind a fallback design?
Three constraints bind a fallback design: preserve quality, control cost, and stay observable. Each one binds only when it is measurable in your deployment, and the published guidance sets no measurement method or threshold for any of them.
Quality floor
The quality floor is the minimum acceptable response quality a fallback may return. Below it, the fallback does not substitute a degraded response for the primary result. A floor can be a score, a pass condition, or a human review step, and the published guidance does not prefer one of those. What matters is that the floor exists and can be measured, because an unmeasurable floor cannot bind a reroute decision. Set it before implementation, then check it in the simulation runs.
Budget
A budget is the maximum cost a fallback chain may consume per request or per period. A backup provider can cost more than the primary, which is why cost control belongs in the contract from the start. The published guidance supplies no amounts, so no dollar figure appears here: MixRoute pricing and the savings calculator are reasonable places to start an estimate. The number you land on carries into every simulation run, since a budget only becomes testable once it is a number.
Observability
The observability constraint tells you when a fallback fires. A design that satisfies it records, per request, which trigger fired, which provider handled the request, and the outcome. Without that record, cost stays invisible and a quality regression gets blamed on the wrong link in the chain. The published guidance prescribes no metric definitions, so the counters you pick are yours. Local simulation, described in the validation section, is the way to check the behavior before production.
None of the three carries a threshold from the published guidance, so the numbers and the definitions are yours, and they belong in the contract where the simulation runs can check them.
What role do circuit breakers and health checks play in fallback?
They help, but neither is the design centerpiece. A circuit breaker watches endpoint health continuously and opens once failures cross a threshold, so an endpoint that is already failing stops dragging down the calls behind it; while the breaker is open, the fallback keeps serving. Published guidance describes the pattern at that level and stops there, leaving the threshold values with you.
Health checking is the input side of the same loop: the endpoint is watched continuously, and the breaker reacts to what the watch reports. In a production setup it is a supporting practice, not the reroute decision itself.
The breaker also sits in the retry path. The TrueFoundry guide on LLM fallback places bounded retries, exponential backoff, and circuit breakers before any reroute when the failure looks transient, which keeps the breaker attached to the retry step and separate from the reroute decision.
How do you validate fallback behavior before production?
Simulate each trigger locally against a mock or test provider, and no production outage is needed to prove the logic. The simulation pushes the same observable signals the gateway sees in production: HTTP status, connectivity or authentication errors, and request timing.
Simulation steps
Run the sequence trigger by trigger, against a mock or test provider on your own machine:
- Stand up a mock provider that can return HTTP 500, 502, 503, and 504, a connectivity failure, a response slower than the timeout, and an authentication error.
- Point the gateway at the mock provider and send normal requests to establish a baseline.
- Force each trigger one at a time and check, per the contract, that the fallback reroutes only on the four evidenced classes.
- Send an HTTP 429 and check that the retry path runs with backoff first, and that the reroute happens only once the limit has not cleared within the retry budget.
- Log the trigger, the reroute decision, and the outcome for each case, so a wrong decision shows up as a line in the run.
The method comes from the Bifrost Mocker demo, a third-party SDK, not from MixRoute telemetry or a customer case study. What a local run shows is narrow: the fallback logic behaves as designed when those signals occur. It says nothing about production latency, 429-rate, failover duration, or savings outcomes.
How should user disclosure and stop conditions be handled in fallback design?
Both are decisions you settle during design, and neither has published guidance behind it. Set a rule for what the user learns when a backup answers, and set stop conditions that end the chain before it runs away.
User disclosure
Disclosure answers a simple question: what does the user find out when a backup provider produces the response? A fallback can change which model or provider answered, so one proposed rule is to surface that a backup handled the request. Published guidance offers no evidence for how or when that should happen, which leaves the mechanism to your deployment. Disclosure also leans on observability: if the request record already names the provider that answered, a disclosure message can be built on top of that record, or you can choose not to disclose at all. Either choice stays a proposed decision.
Stop conditions
A stop condition ends the fallback chain. Candidates include a maximum number of reroutes, a maximum elapsed time, and a ceiling on cost per request. When one fires, the chain returns the best available result or an error and stops. The thresholds are yours, because the published guidance provides no values, and a multi-provider outage is the case that makes them matter: each reroute can add latency and cost while the outage continues.
What stays outside this contract is MixRoute’s own configuration, since none of the triggers, constraints, or stop conditions above map to a MixRoute setting. MixRoute smart routing answers a different question: it selects a model by request complexity from the pool the customer authorizes, which is a cost mechanism and not a failover path. For the cost side, MixRoute pricing and the savings calculator are the reference points. An operator who wants the contract reviewed against their own mix can talk to sales.
FAQ
What is LLM API fallback?
It is a resilience pattern that automatically reroutes a request to a backup model or provider when the primary fails, times out, or is rate-limited, so the caller sees a response instead of an error. The useful part of the definition is the boundary it puts on the trigger list: four classes the gateway can detect, plus a rate limit that survives its retries. A failure you cannot observe at the gateway does not belong on the list, because the gateway has nothing to react to.
How is fallback different from a retry?
A retry re-sends the same request to the same target, while a fallback reroutes it to a backup model or provider. Retries come first and stay bounded, with backoff and a circuit breaker in front of them. The two are answering different questions: a retry asks whether a blip will clear on its own, and a fallback asks whether a different endpoint can answer at all. If a malformed body caused the failure, neither helps, because the backup rejects the same body.
Which errors trigger an LLM API fallback?
The four evidenced classes are network connectivity issues, provider API errors on HTTP 500, 502, 503, or 504, request timeouts, and authentication failures, with a persistent rate limit added once the retry path has had its turn. The line to hold is between failures that are about the endpoint and failures that are about the request. A malformed body or an unsupported parameter fails the same way on a backup, so rerouting it only relocates the error.
Is a circuit breaker required for LLM fallback?
No. A breaker prevents cascade failures by shortening the time you spend calling an endpoint that is already failing, but the fallback contract works without one. The published guidance describes the shape of the pattern, retries with backoff, then a breaker, then a reroute while the breaker is open, and supplies no threshold values. If you run one, the trip point is yours to pick and to write into the contract.
What constraints should a fallback design respect?
Three: keep a quality floor, cap the cost, and stay observable. Each one binds only when it is measurable, which is why a floor with no definition and a budget with no number do not count for much. A practical order is the floor first, so a backup that answers below it never replaces the primary result, then the budget, then the field each request records.
How do I test fallback logic without a production outage?
Run it locally against a mock provider. Stand up a mock that can return HTTP 500, 502, 503, and 504, a connectivity failure, a slow response, and an authentication error, then force one trigger at a time and watch where the request lands. Because you are pushing the same signals the gateway sees in production, the run tells you whether the decision logic matches the contract. It does not tell you anything about production latency or failover duration.
Does published guidance support user disclosure or stop conditions for fallback?
No, neither one is evidenced by published guidance, so both are decisions you make and write into the contract. The practical version is short: pick a way to tell the user that a backup answered, if you choose to tell them at all, and set a cap on reroutes, elapsed time, or cost per request so the chain has somewhere to stop. The cap earns its keep during a multi-provider outage, when each extra hop can add latency and cost.
