LLM API Retry Strategies: Backoff, Jitter, and Retry Budgets

Any production LLM API integration should assume that failures will occur. The question is not whether your application hits a 429 or a timeout, but what it does in the seconds after. A naive retry doubles the load on an already struggling provider. A well-designed retry strategy recovers from transient errors without creating new ones. This guide walks through which failures to retry, how to implement bounded backoff with jitter, how to set retry budgets that prevent storms, and what retries actually cost in real money.

Which LLM API failures are worth retrying

Not every error deserves a retry. The first decision in any retry strategy is classifying failures into three buckets: retryable, failover-worthy, and abort-worthy. Getting this wrong is the most common source of retry-related outages. For the broader failure sequence, see our LLM API failure-handling guide.

Retryable failures: transient and same-provider

Retryable errors indicate a temporary condition that typically resolves within seconds:

• HTTP 429 (rate limit exceeded). The provider is throttling your request tier. Retry after the window expires; see the 429 rate-limit troubleshooting guide for the diagnostic steps.
• HTTP 502 (bad gateway). A proxy or load balancer hiccup. Usually resolves on the next attempt.
• HTTP 503 (service unavailable). The provider is overloaded or under maintenance. Retry with increasing delay.
• HTTP 504 (gateway timeout). The request exceeded the provider’s internal timeout. May succeed with a shorter prompt or a different model.

Failover-worthy failures: sustained and provider-level

Failover-worthy errors suggest the provider itself is degraded, not just your request:

• HTTP 500 (internal server error) in bursts. If you see multiple 500s across different requests in a short window, the provider is likely experiencing an incident. Failover to the next provider in your cascade rather than retrying the same one.
• Connection refused or DNS resolution failure. The provider’s endpoint is unreachable.

Abort-worthy failures: non-retryable and client-side

Retrying an abort-worthy error wastes time and money:

• HTTP 400 (bad request). Your payload is malformed. Fix the request, do not retry it.
• HTTP 401 (unauthorized). Your API key is invalid or expired.
• HTTP 403 (forbidden). Your key lacks permission for this model or endpoint.
• HTTP 404 (not found). The model or endpoint does not exist.
• Context length exceeded. The input is too long for the model. Truncate or split, do not retry unchanged.

The distinction matters because a 429 tells you to wait, a 5xx tells you to try elsewhere, and a 400 tells you to fix your code. Treating all three the same is how applications turn a brief hiccup into a slow denial of service against themselves.

Exponential backoff with jitter: the implementation that works

When you retry, you need a wait strategy. The standard answer is exponential backoff with jitter, but the details determine whether it actually works in production.

Exponential backoff means each retry waits longer than the last. The simplest formula is:

wait_time = min(base_delay * 2^attempt, max_delay)

With a 1-second base delay and a 30-second cap, your retries wait 1s, 2s, 4s, 8s, 16s, 30s. Without jitter, every client that failed at the same instant retries at the same instant, creating a synchronized wave that keeps the provider down.

Jitter randomizes each wait to break synchronization. Three common jitter strategies:

• Full jitter: wait = random(0, base * 2^attempt). Maximum randomization, simplest to implement.
• Equal jitter: wait = base * 2^attempt / 2 + random(0, base * 2^attempt / 2). The formula’s floor is half the exponential wait, so it does not retry too aggressively.
• Decorrelated jitter: wait = min(max_delay, random(base, previous_wait * 3)). Each wait depends on the last, producing less predictable spacing.

Full jitter is the common default for LLM API calls, recommended by the AWS Architecture Blog post Exponential Backoff And Jitter, which concludes that “Equal Jitter is the loser” because it does slightly more client work than full jitter. The AWS SDKs and Tools Reference says the SDK uses exponential backoff with full jitter and prints the delay as random(0, 1) multiplied by the smaller of 20,000 ms and base_delay times 2 to the power of the retry number, checked 2026-08-31. Equal jitter still exists because its formula sets a minimum wait between attempts.

Respect Retry-After

When a provider sends a Retry-After header (common with 429 responses), use that value instead of your calculated backoff. The provider knows its own recovery timeline better than your client does. Retrying before the window expires just earns another 429.

Cap your attempts

Three to five retries is the standard range for API calls. A retry loop with no ceiling is not resilience; it is a slow denial of service against yourself. After hitting the cap, fall through to the next provider or return a structured error to your caller.

Retry budgets: how to prevent a retry storm

Per-request retry limits are necessary but not sufficient. Consider a system handling 10,000 requests per minute. If 5% of requests fail and each retries up to 5 times, you have added 2,500 extra requests per minute to an already struggling provider. The retries amplify the failure instead of recovering from it.

A retry budget caps the total number of retries across a time window, not just per request. Two common patterns:

Per-window budget

Set a maximum percentage of total requests that can be retries. For example: “no more than 10% of requests in any 60-second window may be retries.” Once the budget is exhausted, subsequent failures are returned as errors immediately without retrying. Under that rule your retry traffic stays within a known fraction of your baseline load.

Per-request budget with a global cap

Each request gets up to N retries (typically 3), but a global counter tracks total retries. When the global counter exceeds the budget (say, 500 retries per minute), new failures fail fast. This is simpler to implement than a full per-window budget and catches the same storm scenario.

The budget approach connects directly to cost. Every retry is a request that may be billable. Without a budget, a provider outage that causes 5% failure rate across 100,000 requests can generate 20,000 extra retry attempts, given the four-retry cap used later in this article. With a 10% retry budget, that number caps at 10,000. The difference is not just operational; it is financial.

Handling retries when the response is streaming

Streaming adds a layer of complexity to retries because a failure mid-stream is fundamentally different from a failure before the stream starts.

Pre-stream failures

A connection refused, or a 429 that arrives before any data, is straightforward: retry the full request. The client has not processed any output, so there is nothing to reconcile.

Mid-stream failures

A connection that drops after chunks have arrived, or a provider error during generation, requires a different strategy. You have already received and possibly processed partial output. Simply retrying the full request risks duplicate work: the caller may have already acted on the partial response.

When a mid-stream failure occurs, a retry means re-issuing the generation request. The OpenAI Cookbook page on the seed parameter notes that “determinism is not guaranteed” even with the same seed and parameters, checked 2026-08-31. The second generation therefore produces different text, so partial output from the first attempt cannot be matched against it by comparing content. The mechanism which does work is resuming the interrupted response instead of replaying the request.

For applications that process streaming output incrementally (building a document chunk by chunk, updating a UI in real time), the practical approach is:

• Buffer chunks. Every recovery path starts from the content already received.
• Anthropic. Its streaming documentation recovers an interrupted stream by capturing the partial response, sending a continuation request that carries that content, and resuming from where the stream stopped. For Claude 4.6 and later the partial content is placed in a user message asking the model to continue, while tool use and extended thinking blocks cannot be partially recovered.
• OpenAI. Its Responses API background mode is created with background and stream both true, each streamed event carries a sequence_number you keep as a cursor, and you reconnect to the same response id with starting_after set to that cursor.
• Neither available. Discard the partial output, re-run the whole request, and show the caller nothing until the stream completes.

This is more complex than pre-stream retries, but resuming costs more to implement than a plain retry, and the alternative is a defect the reader sees: the opening of the answer delivered twice in two different wordings, or the partial work discarded.

The cost of retrying: what you might actually pay

Every retry is a full API call. Whether that call is billable depends on the provider’s billing policy, and the policies differ.

Some providers charge for every request that reaches the model, regardless of whether the response was completed successfully. Others do not charge for requests that fail before reaching the model (such as a 429 that is rejected at the rate limiter). The distinction matters: a 429 may be free, but a 502 that reaches the model before the provider’s proxy fails may have already consumed your tokens.

The safe assumption is that any retry is potentially billable. Calculate your retry overhead as:

retry_cost = original_requests * retry_rate * avg_cost_per_request

If you make 100,000 requests per day at $0.01 each, and 5% of requests retry once on average, the potential retry cost is 5,000 × $0.01 = $50 per day. With a retry budget that caps retries at 3% of total traffic, that drops to $30 per day.

If your architecture includes a routing layer, check whether its pricing adds a platform fee on top of provider charges. Include that fee in the retry-cost formula when it exists. Work out what your retry volume adds on top of provider charges. A no-markup plan would not add a platform fee, but confirm the current pricing terms before using that assumption in a budget.

A reference retry policy you can copy

This is a production-ready retry configuration for LLM API calls. Adapt the values to your traffic patterns and provider contracts.

retry_policy = {
  "max_retries": 4,
  "base_delay_ms": 1000,
  "max_delay_ms": 30000,
  "jitter": "full",
  "retryable_status_codes": [429, 502, 503, 504],
  "failover_status_codes": [500],
  "abort_status_codes": [400, 401, 403, 404],
  "retry_budget": {
    "window_seconds": 60,
    "max_retry_percentage": 10
  },
  "respect_retry_after": true,
  "idempotency_key_header": "X-Idempotency-Key"
}

Test cases to validate this policy:

• Single 429: Client waits ~1s (base + jitter), retries once, succeeds. Total: 2 requests.
• Burst of 429s: Multiple requests fail simultaneously. Jitter spreads retries across the window. Retry budget caps total retries at 10% of traffic. No synchronized wave.
• Provider outage (500s): Failover to next provider after 1 attempt (failover_status_codes). No retry on the same provider.
• Invalid request (400): Immediate abort, no retry. Error returned to caller with the provider’s error message.
• Streaming mid-chunk failure: Resume the interrupted response from the cursor, or send a continuation request carrying the partial content, and do not show any output to the caller until the stream completes.
• Budget exhausted: After 10% of traffic is retries, new failures fail fast. No retry attempted.

The policy above handles the failure modes that matter in production. The exact numbers (4 retries, 1s base, 10% budget) are starting points; adjust them based on your provider’s behavior and your cost tolerance.