LLM API Retry Strategies: Backoff, Jitter, and Retry Budgets

Retry 429, 502, 503, and 504. Everything else either goes to another provider or comes straight back to your caller: a burst of 500s is a failover signal, and a 400, 401, 403, 404, or an over-long prompt will fail again in exactly the same way. Wait with exponential backoff plus full jitter, cap attempts at four, and cap total retries at 10% of traffic inside a 60-second window. Every retry may be billable, so the budget is worth settling before you tune the base delay.

Which LLM API failures are worth retrying

Sort each failure into one of three buckets, because everything downstream follows from that: retry the same request when the error is transient, move to a different provider when the provider itself is degraded, and hand the error back to your caller when the request is what’s wrong. Getting this wrong is the most common source of retry-related outages. For the wider failure sequence, see our LLM API failure-handling guide.

Retryable failures: transient and same-provider

Retryable errors point at a temporary condition, one that typically clears within seconds:

• HTTP 429 (rate limit exceeded). The provider is throttling your request tier. Wait out the window, then try again; the 429 rate-limit troubleshooting guide walks through the diagnostics.
• HTTP 502 (bad gateway). A proxy or load balancer hiccuped. Usually clears on the next attempt.
• HTTP 503 (service unavailable). The provider is overloaded or under maintenance. Retry with a longer wait each time.
• HTTP 504 (gateway timeout). The request outlived the provider’s internal timeout. A shorter prompt or a different model may get through.

Failover-worthy failures: sustained and provider-level

Here the provider is degraded, not your request, and a second call to the same endpoint just spends money:

• HTTP 500 (internal server error) in bursts. Several 500s across different requests in a short window point at an incident on the provider’s side. Move to the next provider in your cascade instead of retrying this one.
• Connection refused or DNS resolution failure. The provider’s endpoint is unreachable, so there is nothing left to retry against.

Abort-worthy failures: non-retryable and client-side

Retrying one of these burns a request and buys nothing, because the same body fails the same way:

• HTTP 400 (bad request). Your payload is malformed. Fix the request, then send it.
• HTTP 401 (unauthorized). Your API key is invalid or expired.
• HTTP 403 (forbidden). Your key lacks permission for this model or endpoint.
• HTTP 404 (not found). The model or endpoint does not exist.
• Context length exceeded. The input is too long for the model. Truncate or split it, and do not resend it unchanged.

If your client retries on “the call failed” rather than on the status code, it is resending 400s and 401s along with everything else, and none of those can succeed. That is how a brief hiccup becomes a slow denial of service against your own traffic.

Exponential backoff with jitter: the implementation that works

Wait longer after each attempt, then randomize the wait so your clients stop moving in lockstep. The version that holds up in production is a 1-second base delay that doubles per attempt up to a 30-second cap, full jitter on every wait, and a Retry-After header overriding the calculation whenever the provider sends one.

Exponential backoff means each attempt waits longer than the one before it. The simplest formula:

wait_time = min(base_delay * 2^attempt, max_delay)

With a 1-second base delay and a 30-second cap, your retries wait 1s, 2s, 4s, 8s, 16s, 30s. Without jitter, every client that failed at the same instant comes back at the same instant, and that synchronized wave is what keeps a provider down.

Jitter randomizes each wait to break the synchronization. Three common strategies show up in client libraries:

• Full jitter: wait = random(0, base * 2^attempt). Maximum randomization, minimal code.
• Equal jitter: wait = base * 2^attempt / 2 + random(0, base * 2^attempt / 2). Its floor is half the exponential wait, so it cannot retry as aggressively.
• Decorrelated jitter: wait = min(max_delay, random(base, previous_wait * 3)). Each wait depends on the last, so the spacing is less predictable.

Full jitter is the common default for LLM API calls. It comes from the AWS Architecture Blog post Exponential Backoff And Jitter, which concludes that “Equal Jitter is the loser” because it does slightly more client work than full jitter. The AWS SDKs and Tools Reference says the SDK uses exponential backoff with full jitter and prints the delay as random(0, 1) multiplied by the smaller of 20,000 ms and base_delay times 2 to the power of the retry number. Equal jitter survives in client libraries because its formula guarantees a minimum wait between attempts.

Respect Retry-After

When a provider sends a Retry-After header, which is common on 429 responses, that value beats your calculated backoff. The provider knows its own recovery timeline and your client does not. Coming back before the window closes just earns another 429.

Cap your attempts

Three to five retries is the standard range for API calls. A loop with no ceiling is not resilience; it is your client adding load to a provider that is already struggling. When you hit the cap, fall through to the next provider or return a structured error to your caller.

Retry budgets: how to prevent a retry storm

Per-request limits are not enough on their own. Cap the share of your traffic that can be retries, and the worst case stays bounded even when a provider is failing broadly. A workable starting point is 10% of requests inside a 60-second window.

Picture a service handling 10,000 requests per minute. If 5% of those fail and each one retries up to 5 times, you have added 2,500 requests per minute to a provider that is already struggling. The retries amplify the failure instead of recovering from it, and that is the storm.

A retry budget caps retries across a time window instead of per request. Two common patterns cover production setups:

Per-window budget

Set a ceiling as a percentage of total requests: no more than 10% of requests in any 60-second window may be retries. Once the budget runs out, later failures come back as errors immediately, with no attempt made. Your retry traffic then stays inside a known fraction of baseline load.

Per-request budget with a global cap

Each request gets up to N retries, typically 3, while a global counter tracks the total. When that counter passes the budget, say 500 retries per minute, new failures fail fast. It is simpler to build than a full per-window budget and catches the same storm.

Budgets also cap what a bad day costs you. Every retry is a request that may be billable. A provider outage that fails 5% of 100,000 requests can generate 20,000 extra attempts under the four-retry cap used further down, and a 10% retry budget holds that number at 10,000. The gap shows up on your invoice.

Handling retries when the response is streaming

Where the failure happens inside the stream decides what you can do about it. Before the first chunk arrives you retry the whole request; after chunks have reached your caller, a plain retry is the wrong move and the recovery path is resuming the response.

Pre-stream failures

A connection refused, or a 429 that arrives before any data, is the easy case. The client has processed no output, so retry the full request and nothing needs reconciling.

Mid-stream failures

A connection that drops after chunks have arrived, or a provider error partway through generation, is different. Output has already been received and possibly acted on, so replaying the request risks duplicate work.

Retrying here means re-issuing the generation request. The OpenAI Cookbook page on the seed parameter notes that “determinism is not guaranteed” even with the same seed and parameters. The second generation produces different text, so partial output from the first attempt cannot be matched against it by comparing content. What does work is resuming the interrupted response instead of replaying the request.

For applications that process streamed output incrementally, building a document chunk by chunk or updating a UI as tokens arrive:

• Buffer chunks. Every recovery path starts from the content you already received.
• Anthropic. Its streaming documentation recovers an interrupted stream by capturing the partial response, sending a continuation request that carries that content, and resuming from where the stream stopped. On Claude 4.6 and later the partial content goes into a user message asking the model to continue, while tool use and extended thinking blocks cannot be partially recovered.
• OpenAI. Its Responses API background mode is created with background and stream both true, each streamed event carries a sequence_number you keep as a cursor, and you reconnect to the same response id with starting_after set to that cursor.
• Neither available. Discard the partial output, re-run the whole request, and show nothing to the caller until the stream finishes.

Resuming takes more work to build than a plain retry does. The alternative is the defect your callers notice: the opening of an answer delivered twice in two different wordings, or partial work thrown away.

The cost of retrying: what you might actually pay

Assume every retry is potentially billable and nothing about the invoice will surprise you. Providers differ on failed requests. Some charge for anything that reaches the model, others do not charge for requests rejected before the model sees them, and that difference decides whether a retry storm shows up on your bill.

A 429 stopped at the rate limiter may cost nothing. A 502 that reaches the model before the provider’s proxy fails may have already consumed your tokens.

Calculate the overhead as:

retry_cost = original_requests * retry_rate * avg_cost_per_request

At 100,000 requests per day and $0.01 each, 5% of requests retrying once on average costs 5,000 × $0.01 = $50 per day. With a retry budget that caps retries at 3% of total traffic, that drops to $30 per day.

If a routing layer sits in your architecture, check whether its pricing adds a platform fee on top of provider charges, and put that fee into the formula when it exists. Work out what your retry volume adds on top of provider charges. A no-markup plan would not add a platform fee, but confirm the current pricing terms before you build that assumption into a budget.

A reference retry policy you can copy

This is a production-ready retry configuration for LLM API calls. Adapt the values to your traffic patterns and your provider contracts.

retry_policy = {
  "max_retries": 4,
  "base_delay_ms": 1000,
  "max_delay_ms": 30000,
  "jitter": "full",
  "retryable_status_codes": [429, 502, 503, 504],
  "failover_status_codes": [500],
  "abort_status_codes": [400, 401, 403, 404],
  "retry_budget": {
    "window_seconds": 60,
    "max_retry_percentage": 10
  },
  "respect_retry_after": true,
  "idempotency_key_header": "X-Idempotency-Key"
}

Test cases to validate this policy:

• Single 429: the client waits about 1s (base plus jitter), retries once, succeeds. Total: 2 requests.
• Burst of 429s: several requests fail at once. Jitter spreads the retries across the window, the retry budget caps total retries at 10% of traffic, and no synchronized wave forms.
• Provider outage (500s): failover to the next provider after 1 attempt (failover_status_codes). No retry against the same provider.
• Invalid request (400): immediate abort, no retry. The error goes back to the caller with the provider’s error message.
• Streaming mid-chunk failure: resume the interrupted response from the cursor, or send a continuation request carrying the partial content, and hold all output back from the caller until the stream completes.
• Budget exhausted: once retries reach 10% of traffic, new failures fail fast and no attempt is made.

The numbers in the policy (4 retries, a 1s base, a 10% budget) are starting points. Move them based on how your providers behave and how much retry spend you can tolerate.