How to Handle LLM API Failures: Rate Limits, Outages, and Failover (2026)

Every major LLM provider went down in 2026. OpenAI’s API failed globally on July 25, taking ChatGPT, the developer API, and Codex with it, cause unidentified for hours. Anthropic logged ten separate disruptions across twelve days in June, having acknowledged its infrastructure was stretched by unprecedented demand. Provider uptime around 99.5% sounds reassuring until you count the hours: that is roughly 41 hours a year your app is down if it depends on one provider. The question is not whether your provider fails. It is whether your app fails with it.

This guide covers the three ways an LLM call fails and the three layers of defense that keep your product running when a provider does not.

The three ways an LLM call fails

Failures are not one thing, and each mode wants a different response.

Rate limits (HTTP 429). You are sending requests faster than your tier allows, or the provider is shedding load. The request is rejected, often with a Retry-After header telling you how long to wait. This is the most common failure and the most recoverable.

Outages and overload (HTTP 5xx). The provider itself is degraded. A 500 is a server error, a 503 means overloaded, and during a real incident you get these in bursts across every request. This is what happened to OpenAI on July 25 and to Anthropic repeatedly through June. No amount of retrying fixes a provider that is down.

Timeouts and latency. The request does not fail, it just does not come back in time, or comes back so slowly it breaks your user experience. A model under load can take many times its normal latency. A slow success can hurt more than a fast failure, because you pay for it and your user still waits.

Knowing which mode you are in decides the fix. You retry a 429. You fail over on a 5xx. You cap a timeout. Treating all three the same is why apps fall over.

Do the math on “99.5% uptime”

Providers quote uptime as a percentage because a percentage sounds good. Convert it to hours and the picture changes.

If your product is hardcoded to a single provider running at 99.5%, you have accepted roughly a full working week of downtime a year, scheduled entirely by someone else, usually with no warning. For an internal tool that might be fine. For anything customer facing, it is a business risk sitting in a config file.

Here is the part that reframes the whole problem. Two independent providers, with automatic failover between them, combine their availability. Two providers each at 99.53% give a combined availability above 99.99%, which cuts expected downtime from 41 hours a year to roughly 12 minutes. That is a 200x reduction, and it is the entire case for not depending on one provider.

Layer 1: retries with exponential backoff

The first and cheapest defense. Most transient failures, a momentary 429 or a brief blip, resolve if you simply try again in a moment. But naive retries make things worse, so four rules matter.

Back off exponentially. Wait 1 second, then 2, then 4, then 8. Retrying immediately hammers a provider that is already struggling and gets you rate limited harder.

Add jitter. Randomize each wait a little. Without jitter, every client that failed at the same instant retries at the same instant, and the synchronized wave keeps the provider down. This is the thundering herd, and jitter is the one-line fix.

Respect Retry-After. When the provider tells you how long to wait, obey it instead of guessing. Retrying before that window just earns another 429.

Cap your attempts. Three to five retries, then give up and fall through to the next layer. A retry loop with no ceiling is not resilience, it is a slow denial of service against yourself, and it runs up the bill while it does it.

That last point connects to cost. Every retry is a full charge if the call reaches the model before failing. An app that retries aggressively can quietly bill 1.4 or more calls per request, so log your real attempts-per-request ratio and keep the cap tight.

One reliability bonus worth knowing: on providers that support prompt caching, cache hits are not counted against your rate limits. A well cached stable prefix buys you rate-limit headroom as well as cost savings, which means fewer 429s to handle in the first place.

Layer 2: timeouts and circuit breakers

Retries handle brief failures. They are exactly the wrong tool for a sustained outage, because retrying a provider that is down for twenty minutes just wastes twenty minutes of your users’ time and your money.

Set an explicit timeout. Decide the longest you will wait for a response and enforce it. A request that has not returned by then is a failure, even if the provider would eventually answer. Match the timeout to the experience: shorter for interactive features, longer for background jobs.

Add a circuit breaker. When a provider fails repeatedly in a short window, stop sending it traffic entirely for a cooldown period, then test with a single request before resuming. This does two things. It stops you burning time and money on calls that will fail, and it lets a struggling provider recover instead of being hammered by your retries.

The circuit breaker also solves a subtler problem the July 25 outage put on display: a status page can lag reality, at points showing systems operational while the incident is still open. Your own error rate is a faster and more honest signal than any status page. A circuit breaker acts on what is actually happening to your requests, not on what a dashboard claims.

Layer 3: multi-provider failover

This is the real fix, and the other two layers exist mostly to feed into it. When a provider is rate limiting you or is down, route the request to a different provider that is not. Your effective availability stops being any single provider’s number and becomes the combined number, which is where the 200x reduction above comes from.

Anthropic’s own guidance during capacity events points the same direction: switch to a different model tier when your primary is constrained. Failover generalizes that across providers.

The thing that makes this practical in 2026 is the OpenAI compatible API standard. Most providers now expose an endpoint that speaks the same request and response format, so failing over from one to another is a change of base URL and model string, not a rewrite. The pattern:

providers = [
    ("primary",  "https://api.provider-a.com/v1", "model-a"),
    ("fallback", "https://api.provider-b.com/v1", "model-b"),
]

for name, base_url, model in providers:
    try:
        return call(base_url, model, messages, timeout=20)
    except (RateLimitError, ServerError, Timeout):
        continue          # circuit breaker decides when to skip a provider outright
raise AllProvidersFailed()

Two things to get right when you build it:

Idempotency and deduplication. If a request times out but actually succeeded on the provider’s side, a naive failover runs it again and you get, and pay for, duplicate work. Use idempotency keys where the provider supports them, and design the calling code so a retry of a completed request is safe.

A quality-matched fallback. The fallback model should be good enough for the task, not just any model that happens to be up. Falling over from a frontier model to a weak one keeps you online but degrades output, which for some tasks is worse than a clean error. Pick a fallback per task, the same way you route per task for cost.

The failures that do not throw

Not every failure raises an exception, and the silent ones are the ones that reach users.

A floating model alias. If your model string is a floating alias rather than a pinned snapshot, the provider can move it to a new model under you. Nothing errors, but behavior, tokenizer, and cost all shift. Pin the snapshot in production.

Degraded quality during load. Some providers serve a faster, lower-quality path under heavy load rather than failing outright. Your calls succeed, your outputs quietly get worse, and no error tells you. This is why an evaluation set matters even for reliability, not just cost.

Partial outages. One model is down while others are fine, or one region fails while another holds. A blanket “is the provider up” check misses these. Health-check the specific model and path you actually use.

The reliability checklist

A production LLM integration should have all of these before it matters, not after the first outage.

1. Retries with exponential backoff, jitter, Retry-After compliance, and a hard cap.
2. Explicit timeouts matched to each feature’s tolerance.
3. A circuit breaker that trips on your own error rate, not a status page.
4. A second provider reachable behind the same interface, with a quality-matched fallback per task.
5. Idempotency so a failover cannot double-charge you.
6. Pinned model snapshots in production.
7. Monitoring on error rate, latency, and cost per task, alerting you before users do.

Where a gateway fits

Layers 1 and 2 live in your application code and always will. Layer 3 is the one that is genuinely painful to build yourself, because multi-provider failover means integrating, authenticating, and monitoring several providers, normalizing their differences, and maintaining health checks across all of them. That work is why most teams skip it and stay single-provider, right up until the morning their provider goes down.

An AI API gateway does layer 3 for you. Every provider sits behind one OpenAI compatible endpoint, and when one is rate limiting or down, the gateway routes to a healthy one automatically, with no change to your code. Your application makes one call to one endpoint, and the failover happens underneath it. To be clear about the boundary: a gateway does not replace your retries, timeouts, and idempotency, which are still yours to own. What it removes is the multi-provider integration burden that makes real failover impractical to build by hand.

MixRoute is built for exactly this: one OpenAI compatible API across 200+ models with automatic failover when a provider drops, and zero markup on provider pricing. The reliability you would otherwise assemble from several accounts and a pile of health-check code becomes a single integration. Smart routing, which picks the best model per request on quality, cost, and latency, is coming.

FAQ

How do I handle rate limit errors from an LLM API? Retry with exponential backoff and jitter, and obey the Retry-After header when the provider sends one. Cap retries at three to five attempts, then fall through to a fallback provider. Prompt caching also helps, because on supporting providers cache hits do not count against your rate limit, reducing 429s at the source.

What happens to my app when OpenAI or Anthropic goes down? If you depend on a single provider, your AI features go down with it, as many apps discovered during OpenAI’s July 25, 2026 outage and Anthropic’s repeated June disruptions. The fix is multi-provider failover: when your primary is down, requests route automatically to a different provider that is up.

How much downtime does 99.5% uptime actually mean? About 3.4 hours a month and 41 hours a year. Quoted as a percentage it sounds negligible, but converted to hours it is close to a full working week of annual downtime for a single-provider app, scheduled by the provider and usually unannounced.

What is the difference between retrying and failover? Retrying sends the same request to the same provider again after a wait, which fixes brief, transient failures. Failover sends the request to a different provider, which is what you need during a sustained outage. Retrying a provider that is down just wastes time. Use retries for blips and failover for outages, with a circuit breaker deciding when to switch.

Does multi-provider failover improve reliability that much? Yes, because independent providers combine their availability. Two providers each at 99.53% reach a combined availability above 99.99% with automatic failover, cutting expected downtime from roughly 41 hours a year to about 12 minutes. The providers have to be genuinely independent for the math to hold.

How do I add failover without rewriting my app? Use the OpenAI compatible API standard that most providers now support, so switching providers is a change of base URL and model string rather than a new integration. An AI API gateway takes this further by putting every provider behind one endpoint and handling the failover automatically, so your code calls a single endpoint and never sees the switch.

The bottom line

Every provider fails, and 2026 proved it across the board. Reliability is not hoping your provider stays up, it is designing for the certainty that it will not. Retry the transient failures with backoff and a cap, time out and circuit-break the sustained ones, and fail over to a second provider for the outages that retries cannot fix. Convert uptime to hours before you decide a single provider is good enough, because 99.5% is a working week a year.

MixRoute puts every major model behind one OpenAI compatible endpoint with automatic failover and zero markup, so multi-provider reliability is a single integration instead of a pile of health-check code you maintain forever. Start building on MixRoute