429, 5xx, and Timeouts: Engineering Practices for LLM Gateway Failover

Classify first, then decide how to handle it

'Retry every error three times' is the most common wrong approach. Different error codes have completely different semantics, and the handling strategy must differ too. The table below is the strategy our gateway actually uses, and it can serve as a reference for your own implementation.

• Connection failure / DNS / TLS: switch to the backup channel immediately.
• Timeout (output has not started yet): switch to the backup channel, at most 1–2 times.
• HTTP 429: cool down briefly, then switch channels, and lower that channel's weight at the same time.
• HTTP 5xx: count it in the failure window; consecutive failures trigger the circuit breaker.
• 401 / 403: credential or permission problem; disable that credential and alert, do not retry.
• Insufficient balance: pause that channel and alert.
• 400 parameter error: after normalization, return it directly to the customer; do not retry, do not bill.
• SSE interrupted after partial output: return upstream_stream_error; by default do not retry.

Why an interrupted SSE stream cannot simply be retried

Once a streaming response has started outputting tokens to the client, it means the upstream has already produced output (and will bill for it). If the gateway quietly switches to another channel and starts over, the user sees duplicated content, and you pay for both outputs.

The correct approach is to treat this half-finished stream as a partial success: return an error code telling the client 'upstream interrupted', and let the business side decide whether to show the existing content plus a retry prompt, or start over entirely.

Circuit breaker: the layer more important than retries

Retries handle a single request; the circuit breaker judges the overall state over a period of time. When a channel's consecutive failures reach a threshold, mark it 'open', stop sending traffic to it for a while, then let a small amount of probe traffic through (half-open), and after several successful probes fully restore it (closed).

Without a circuit breaker, retries will crush an already-dead channel even harder and cause the customer's average latency to spike.

Retry idempotency and billing

• Non-streaming requests: after retries, bill only the usage of the final successful attempt.
• Streaming requests: the attempt that already output tokens really incurs upstream cost and must be recorded; you cannot wipe the ledger clean just because of a retry.
• Assign a request_id to each user request; all retries share it, so you can fully replay later 'why this channel was chosen'.
• Refunds/adjustments must go through ledger entries; you cannot directly change the balance.

What Admesh already does

The strategy above is the default behavior in the Admesh gateway: automatic switching among multiple upstream channels for the same public_model, every routing decision is persisted, and customers can see request_id, the model actually used, and channel health in their usage records; error codes are returned in a unified format (error.type / error.code / request_id).

It is worth noting: routing only happens within the channel pool of the same public_model; it will not switch you to another vendor's model just because you requested anthropic/xxx—cross-model substitution directly changes output quality, and that is a decision we cannot make for you.

View the error code reference →