Why Do LLM API Token Limit Failures Happen and How to Fix Them

A token limit failure comes from one of two causes, and the fixes point in opposite directions. Either your traffic emptied a rate-limit bucket, which returns an HTTP 429 that clears if you wait, or one request is larger than the model’s context window, which fails the same way on every retry. Read the error code before you retry anything, because some 429s mean the credit or spend limit ran out and waiting never clears them. For a size rejection, shrink the input or send it to a model with a bigger window, such as claude-sonnet-5-5 at 1M tokens instead of grok-4.7 at 500K.

What is an LLM API token limit failure?

A token limit failure is a request the API rejects or cuts short because token usage passes an allowance the provider enforces, and there are two allowances. The first is a rate quota, stated in tokens per minute (TPM) and requests per minute (RPM), which caps aggregate traffic from an account or workspace. The second is the context window, which caps a single request and its completion together. Rate-limit failures come back as a 429 and clear as the bucket refills. Anthropic’s API uses a token bucket, so capacity is continuously replenished up to the maximum rather than reset at fixed intervals.

Not every 429 is a rate limit. OpenAI returns 429 with credit_balance_exhausted when prepaid credit is gone, with organization_spend_limit_exceeded or project_spend_limit_exceeded when a spend limit is hit, and with organization_usage_limit_exceeded when an OpenAI-assigned usage limit is reached. Anthropic returns a 429 of type rate_limit_error when a monthly spend cap is reached, with enforced_spend_limit_reached in error.details.error_code and no retry-after header. Retrying those does not restore access. The context-window failure is different in kind: the payload is over the size the model accepts, and no amount of waiting changes that.

The two classes of token limit failure

From the caller’s side the two classes look alike, but their remedies point in opposite directions. Transient throttling returns HTTP 429 while the request size sits within limits. Deterministic size rejection returns an error after the payload is measured against the context window. A third, related manifestation is output truncation, where generation ends at the output token ceiling instead of reaching a natural finish. To place a failure in the wider field, read a taxonomy that names every LLM API failure class.

What causes LLM API token limit failures?

Throttling comes from aggregate pressure on the rate quota, and size rejection comes from an assembled context that has grown past the model’s limit. Each class has its own cause, and the cause decides which remedy applies.

Aggregate rate pressure drains the bucket

Picture a bucket that refills continuously. Every request takes tokens out, and when a request arrives to an empty bucket the API answers 429. Parallel agents draw from the same bucket, so a burst drains it faster than the refill rate replaces it. Nothing is wrong with the request itself, which is why the identical request succeeds once the bucket refills. Anthropic measures RPM, input tokens per minute (ITPM) and output tokens per minute (OTPM) per model class, and notes that 60 RPM can be enforced as 1 request per second, so a short burst can trip a limit that looks safe on a per-minute average.

Two other things drain the bucket early. A sharp ramp-up can trigger an acceleration limit even when the totals look fine: OpenAI’s slow_down error can appear within the RPM and TPM limits, and its rule of thumb is to grow traffic by no more than 50% every 15 minutes once you reach 1 million input tokens per minute. And on OpenAI the rate limit is calculated from the larger of max_tokens and an estimate based on the character count of the request, so an oversized max_tokens uses up bucket you never needed. Anthropic evaluates OTPM on tokens actually generated, and max_tokens does not factor in.

Accumulated context crosses the context window

The deterministic class comes from the assembled context. Everything in the request counts toward the window: the system prompt, every message including tool results, images and documents, your tool definitions, and the output the model generates, thinking included. A coding agent that grounds itself in a repository keeps adding file paths and file contents, and every file pushes the total higher. When the input alone passes the window, Anthropic returns a 400 invalid_request_error with the message “prompt is too long”.

The ceiling belongs to the model the request goes to. Say an agent has collected a 600K-token prompt. It fits claude-sonnet-5-5, which lists a 1M context window and 128K max output in MixRoute’s catalog, and grok-4.7 rejects it, because that model lists 500K. The table shows the limits for a few current models, as listed on their MixRoute model pages on 2026-10-19. An older rule of thumb that Haiku is the 200K model no longer holds: claude-haiku-5-5 lists 1M.

On Claude 4.5 models and newer, a request whose input plus max_tokens goes over the window is still accepted, and generation stops with stop_reason model_context_window_exceeded when it reaches the limit. So the same overflow shows up as a 400 or as a truncated 200, depending on which part of the budget crossed the line. For the mechanics of the ceiling itself, read an explanation of how context windows set the single-request ceiling.

How do you detect a token limit failure and output truncation?

Detection is a monitoring job: count the 429s by error code, watch the remaining-token headers, and compare each prompt against the context window of the model it is going to. Those signals show whether pressure is aggregate or payload sized, and that answer picks the remedy.

Monitoring signals that separate the classes

Count HTTP 429 responses by error code and watch the rate over time. A rising count of rate-limit codes points to a bucket emptying under aggregate pressure. Credit and spend-limit codes point somewhere else: a billing limit that waiting will not clear. Anthropic’s spend-cap 429 has the same rate_limit_error type as a real rate limit, so tell them apart by error.details.error_code and by the missing retry-after header. Pair the count with token spend per key or tenant to see which traffic source is draining the quota.

Add the remaining-token headers to forecast the next throttle before it lands. OpenAI sends x-ratelimit-remaining-tokens and x-ratelimit-reset-tokens. Anthropic sends anthropic-ratelimit-tokens-remaining, plus separate input and output variants. The size class shows up differently: a 400 that says the prompt is too long, or a response that ends with model_context_window_exceeded. Compare the prompt’s token count against the model’s context maximum, because a request at or above the maximum is a size rejection by definition. For a fuller signal set, read the metrics that make LLM API observability actionable.

Verifying output truncation

Output truncation needs its own check, because it arrives on a response that looks successful. Read the truncation value your API defines, since each API names it differently:

• finish_reason length on OpenAI Chat Completions.
• status incomplete with incomplete_details.reason max_output_tokens on the OpenAI Responses API.
• stop_reason max_tokens on Anthropic Messages.
• finishReason MAX_TOKENS on Gemini.
• stopReason max_tokens on Bedrock Converse.

Anthropic’s model_context_window_exceeded also marks a truncated answer, and Anthropic’s own guidance is to treat that response as truncated. Client tools print their own wording for the same event, for example “model reached the maximum output token limit before completing the turn”. Those strings come from the client, so look underneath for the API’s marker.

Why is your LLM API token count higher than expected?

The count is higher than your estimate when the request carries more than the text you wrote, or when the model’s tokenizer counts differently from the one you estimated with. Six things account for most of the gap, and each one shows up in the usage block of the response.

• Parts of the request you did not count. The system prompt, tool definitions, tool results, images and documents all count toward the window, and so does the output.
• Reasoning tokens. OpenAI’s reasoning tokens are invisible in the text, occupy the context window and are billed as output tokens. They are visible under output_tokens_details, and OpenAI says they can run from a few hundred to tens of thousands. On Anthropic, thinking tokens are a subset of max_tokens, are billed as output and count toward rate limits. On current Opus, Sonnet, Haiku and Fable models the API also keeps earlier thinking blocks by default, so they come back as input on later turns.
• A different tokenizer. Anthropic states that Claude 4.7 and later models produce approximately 30 percent more tokens for the same text than earlier models, and tells you to recount prompts against the model you plan to use. A count taken on an older model understates the new one. What a token is explains why the same sentence has different counts on different models.
• Cached prefixes split across fields. With prompt caching, Anthropic reports input across input_tokens, cache_read_input_tokens and cache_creation_input_tokens, and all three count toward the window. A 200K-token cached document plus a 50-token question shows input_tokens: 50 while the real input is 200,050.
• Resent history. Even with previous_response_id, OpenAI bills all previous input tokens in the chain as input tokens.
• Calls you did not make by hand. A compaction call is billed and rate-limited like any other request, and Anthropic reports it as the compaction entry in usage.iterations, with zero in the top-level fields. Retries multiply calls too: OpenAI advises disabling SDK retries or counting them in your own limits so nested retry loops do not multiply requests.

Price can also rise with no change in the count. On MixRoute’s catalog, claude-haiku-5-5 bills prompts above 100K tokens in a long-context tier at $0.50 input and $2.50 output per 1M tokens, against $0.10 and $0.50 below it, and gpt-6.1-sol moves from $2 to $4 input above 272K. A prompt that grew past a tier boundary after a deploy costs more per token with the same behavior.

To find where a sudden jump came from, work through the response data:

1. Log the full usage object for every response, including cache fields and reasoning detail.
2. Add up every input field and every output field. For an Anthropic conversation that was compacted, sum across usage.iterations.
3. Compare the sum with your pre-send count. Anthropic’s token counting endpoint returns an estimate that can differ by a small amount, so a large gap points to one of the causes above.
4. Check what changed between the cheap day and the expensive day: the model ID, thinking settings, tool list, or the amount of history sent back.

How do you fix a token limit failure?

Two remediation tracks exist, and you apply exactly one of them, chosen by the class you detected. A rate-limit 429 responds to backoff and retry, because the bucket refills and a delayed request can go through. A credit or spend-limit 429 does not. Size rejection repeats for a plain reason: the same payload meets the same model’s maximum, so the outcome changes only when the input shrinks or the request goes to a model with a larger context window.

Backoff and retry for the throttling track

Read the error code first, and retry only the rate-limit codes. Credit, spend-limit and usage-limit 429s need someone to add credit or raise the limit, so surface them to a human instead of retrying. For rate-limit 429s, wait at least as long as Retry-After says, since Anthropic notes that earlier retries will fail, and add a small random delay so clients do not retry in lockstep. When the header is missing, fall back to exponential backoff with jitter. Cap both the number of attempts and the total time spent retrying, so requests do not loop against an empty bucket.

Reduce aggregate pressure in parallel. Flatten bursts and move non-interactive work to OpenAI’s Batch API, which runs without touching your synchronous rate limits. On Anthropic, cached input read from the cache does not count toward ITPM for most models, so caching repeated prefixes raises your effective throughput. Leave the payload alone on this track, because the problem is when the request arrives. Retry strategies with backoff, jitter and retry budgets covers the loop in code.

Input compaction for the size track

When the class is deterministic size rejection, compact the input. Exclude file paths the task does not need, trim conversation history to recent turns, drop attached files that carry no task value, and split one oversized task into several requests that each fit the context window. If your payload is a repository dump, the file paths are the cheapest thing to cut first. A payload above the context maximum is rejected on size, so the next attempt has to be smaller, or it has to go to a model whose window fits it. For the full sequence, read a production checklist for handling LLM API failures.

The table below classifies the three manifestations and their decision rules.

What does “compaction failed: summarization failed: generation hit the token cap” mean?

It means the step that writes the summary stopped at its own output limit before it finished. Claude Code prints this message, and Anthropic does not document the string. A Claude Code troubleshooting guide reads it as the summarizer running out of output room, which is observed behavior and not a published contract. The Claude API has the same failure with documented handling, and it is output truncation applied to the summary call.

When you request a summary through Anthropic’s on-demand compaction, the response is a 200 with empty content if the summary call did not end normally, so check stop_reason before you look for the block. Two values map to this error:

• max_tokens means the summary was cut off. Resend with a larger max_tokens. That value caps the whole call, thinking included, so Anthropic’s guidance is to allow several thousand tokens.
• model_context_window_exceeded means there was no room for the summarization prompt. Resend with fewer messages or shorter instructions.

The second case has a harder form. The request that summarizes the conversation is as large as the conversation, so a session that already passed the window cannot be rescued by compacting it. An open-source agent’s issue tracker describes exactly that: a session at 620,332 context tokens on a 272k window that /compact could not bring back. Anthropic’s advice is to compact before you outgrow the window, not after.

On the API you can keep working without a summary and compact later. The summary call is billed whether or not a summary comes back, and it appears in usage.iterations. In a tool you do not control, compact earlier, while the session is well under the window. The same Claude Code guide suggests running /compact with a focus, so the summary has less to cover.

How do you prevent token limit failures with pre-send budgeting?

Prevention moves the work to before the send. Estimate the prompt’s token count at assembly time, reserve output headroom inside the context window, and refuse to forward any prompt whose estimated total exceeds the limit. A request that is already over the limit must not leave the client, because sending it only completes a round trip that fails deterministically.

Estimate prompt tokens before the request leaves

Count the prompt against the model it is going to. Anthropic’s token counting endpoint takes the same structured input as a message request, including system prompts, tools, images and PDFs, and returns the input token count. It is free to use, has its own requests-per-minute limits that are independent of message creation, and returns an estimate that can differ from the real count by a small amount. It rejects a few inputs the Messages API accepts, such as server tools and the MCP connector, so for those requests read the usage object in the response instead. For OpenAI models, the tokenizer tool built on the tiktoken library estimates a string.

Then reserve output headroom in the same check. The window covers the prompt and the completion together, thinking included. Take a model with a 500K window, such as grok-4.7, and a max_tokens of 32,000: the prompt budget is 468,000 tokens, and anything larger should be trimmed or sent elsewhere. OpenAI recommends reserving at least 25,000 tokens for reasoning and output when you start with a reasoning model, then adjusting once you see how many reasoning tokens your prompts use. Without a reservation, a prompt that fits exactly can still be cut off at the output ceiling.

Why post-response accounting misleads

Post-response accounting tells you what already happened. During an error storm the usage totals also include calls that returned nothing useful: an incomplete OpenAI response can bill input and reasoning tokens with no visible answer, and an Anthropic summary call is billed even when no summary comes back. Pre-send estimation shifts the check earlier, and the usage field in the response then confirms what the estimate predicted. When the two disagree by a wide margin, go back to the list of causes for a count that runs high.

Compaction has to preserve the parts of the context the task depends on: keep the instruction, the latest user request, and the schemas or facts the task references, and cut only what the task does not need. Anthropic notes that images, documents and fetched URLs inside summarized messages are gone once the summary replaces them, so restate or re-upload anything a later turn still needs.

For the transient class, prevention means pacing traffic under the quota: batch non-interactive work, space requests below the TPM ceiling, and watch the remaining-token headers so the bucket does not run dry. Spikes still happen, and that is what the retry track absorbs.

How does the MixRoute LLM API gateway handle token limit failures?

MixRoute sits as an LLM API gateway between your application and upstream providers, with one OpenAI-compatible base URL, https://api.mixroute.ai/v1. When a request is too large for one model, you send it to a model with a larger window by changing the model ID. The model catalog lists the context length and max output of each model, so you can check the numbers before you switch.

Smart Routing works by having an in-house routing model judge each request’s complexity and task type, then selecting a model from the pool the customer authorizes; the triage step runs in memory with a 1M-token context window and is cleared immediately afterwards. See how Smart Routing selects models. The full context passes through untouched: no rewriting, no compression, no truncation. Compaction therefore stays the caller’s job, and the pre-send budget described above is still work the application must do. Each key can carry its own spend budget, and the key stops when the budget reaches zero. See the pricing structure.

Start routing your LLM traffic through a gateway that routes on complexity and task type.