Fallbacks and Retries
LLM APIs fail. Rate limits hit, servers go down, and timeouts happen. Fallbacks and retries are the engineering patterns that ensure your users always get a response even when the primary model is unavailable.
Why Does This Exist?
OpenAI's API went down three times in a single week in early 2024. Anthropic's Claude API hit capacity errors during peak hours for months after launch. No external LLM API guarantees 100% uptime, and even self-hosted models fail when GPUs overheat or deployments go sideways.
If your application doesn't have a plan for these failures, users hit cryptic errors and leave. With fallbacks and retries, the same failure is invisible to the user — they just get an answer, maybe 200 milliseconds slower.
Think of It Like This
Think of It Like This
You're calling a customer service line. It's busy. You don't immediately hang up and give up — you try again in a few seconds. If it's still busy after three tries, you call an alternative number. If that number is also down, you leave a voicemail (degraded mode). At no point do you walk away without doing anything. Fallbacks and retries are exactly this behaviour, implemented in code.
Retries with Exponential Backoff
A naive retry — immediately re-sending the failed request — makes things worse. If an API is overloaded, hammering it with retries increases the load and delays recovery for everyone. The correct pattern is exponential backoff: wait 1 second before the first retry, 2 seconds before the second, 4 seconds before the third. Each wait roughly doubles.
Add jitter — a random offset of ±20% — to prevent a thundering herd of simultaneous retries after a shared outage all firing at exactly the same moment.
import timeimport randomimport openai
def call_with_retries(prompt: str, max_retries: int = 3) -> str: for attempt in range(max_retries): try: response = openai.chat.completions.create( model="gpt-4o-2024-11-20", messages=[{"role": "user", "content": prompt}], timeout=30.0, ) return response.choices[0].message.content except openai.RateLimitError: # Rate limits: always retry with backoff wait = (2 ** attempt) + random.uniform(-0.3, 0.3) time.sleep(wait) except openai.APIStatusError as e: if e.status_code >= 500: # Server errors: retry wait = (2 ** attempt) + random.uniform(-0.3, 0.3) time.sleep(wait) else: # 4xx client errors: don't retry — the request itself is broken raise raise RuntimeError("All retries exhausted")Only retry on transient errors (5xx server errors, rate limits, network timeouts). Never retry on 4xx client errors — if your request was malformed, retrying it will just get the same error faster.
Fallbacks
When retries are exhausted, a fallback switches to a different model or provider. A well-structured fallback list might be:
gpt-4o(primary, fastest)claude-3-5-sonnet(fallback 1, different provider)gpt-4o-mini(fallback 2, cheaper, always available)- Cached response or graceful error message (degraded mode)
The fallback is not a retry — it's a different endpoint. This protects against provider-level outages, not just request-level failures.
Idempotency: The Hidden Danger
Retries create a subtle bug for operations with side effects. If your LLM call triggers a tool that sends an email or charges a credit card, a retry on a timed-out request might succeed on the second attempt — but the first request might have already succeeded on the server side, just without sending back a response. Now you've sent two emails.
The fix is an idempotency key: a unique ID per logical request that the server uses to deduplicate. OpenAI's API supports this via the Idempotency-Key header. For tool calls with side effects, always generate and pass one.
Degraded Mode
When all fallbacks are exhausted, don't crash. Return a meaningful degraded response: a cached answer from a recent similar query, a "we're experiencing high load" message with a retry suggestion, or a simplified answer from a rule-based system. Users tolerate slow or partial responses far better than blank pages.
Watch Out For
Watch Out For
Retrying on content policy violations. If the API returns a 400 because your prompt violated the content policy, retrying is both pointless and wasteful — the same prompt will hit the same policy every time. Build a classification step that catches these errors before the retry loop, logs them for review, and returns an appropriate user-facing message without wasting three retry attempts.
The Quick Version
- Retries handle transient failures; fallbacks handle provider outages. You need both.
- Use exponential backoff with jitter for retries — never immediate re-sends.
- Only retry transient errors (5xx, rate limits). 4xx errors mean the request is wrong.
- For side-effectful tool calls, pass an idempotency key to prevent duplicate actions.
- Always define a degraded mode so users get something useful even when all fallbacks fail.
What to Read Next
rate-limits-and-quotas— The source of most 429 errors, and how to structure your request queue to avoid triggering them.llm-observability— How to trace failed requests through the retry chain so you can tell which fallback was triggered and why.