The Retry and Fallback Pattern for Resilient LLM Apps
Production LLM services fail in ways traditional APIs rarely do. A model endpoint can time out under load, return a 429 rate-limit error, hallucinate malformed JSON, or simply go dark during a provider outage. If your application treats a single model call as a guaranteed success, one bad response can cascade into a broken user experience. The LLM fallback pattern — combined with disciplined retries, timeouts, and graceful degradation — is how resilient teams keep AI features online even when individual components misbehave.
This guide walks through the concrete building blocks: designing timeouts, structuring retries safely, chaining model fallbacks, and degrading gracefully when everything upstream is failing.
Why LLM Calls Need Special Resilience Handling
Unlike a database query that either succeeds or throws a clear error, LLM calls introduce several unique failure modes:
Latency variance
The same prompt can return in 800ms or 25 seconds depending on output length, provider load, and whether streaming is enabled. Long tail latency is normal, not exceptional.
Transient provider errors
Rate limits (HTTP 429), server overload (503), and capacity errors are common on shared inference infrastructure. Most are recoverable within seconds.
Semantic failures
The request succeeds at the HTTP level, but the content is wrong: invalid JSON, a refusal, or a response that fails your schema validation. These need retry logic that inspects the payload, not just the status code.
A robust design assumes any single call can fail and plans the recovery path in advance.
Step 1: Set Aggressive, Purposeful Timeouts
Never rely on default SDK timeouts, which are often several minutes long. A hung request holds connections, threads, and user patience hostage.
Set timeouts based on the use case, not the worst-case model behavior:
Interactive chat: Use streaming and set a time-to-first-token timeout (for example, 3–5 seconds). If the first token doesn't arrive quickly, fail fast and move to a fallback.
Background batch jobs: A longer overall timeout (30–60 seconds) is acceptable because no human is waiting.
Structured extraction: Cap the request to a tight budget since output length is bounded and predictable.
Distinguish between a connection timeout, a time-to-first-byte timeout, and a total request timeout. Configuring all three prevents a slow provider from silently degrading your service.
Step 2: Retry Transient Failures — Carefully
Retries are powerful but dangerous if applied blindly. Retrying a non-idempotent operation or hammering an already-overloaded endpoint makes outages worse.
Retry only what should be retried
Retry on timeouts, 429s, 500s, 502s, 503s, and connection resets. Do not retry on 400s (bad request), 401/403 (auth), or content-policy refusals — those won't fix themselves.
Use exponential backoff with jitter
Fixed retry intervals cause synchronized retry storms across many clients. Exponential backoff with randomized jitter spreads the load. A practical policy: base delay of 500ms, doubling each attempt, with random jitter added, and a maximum of 2–3 attempts before escalating.
Respect Retry-After headers
When a provider returns a 429 with a Retry-After value, honor it. Ignoring it wastes your retry budget and can extend rate-limit penalties.
Keep retries idempotent
If your call has side effects — writing to a database, charging usage, sending a message — make sure a retried request doesn't duplicate them. Use idempotency keys where the provider supports them.
Step 3: Chain Model Fallbacks
Retries handle transient blips against a single model. The LLM fallback pattern handles sustained failure by routing to an alternative. Build an ordered chain where each tier is tried only after the previous one exhausts its retries.
Common fallback strategies
Cross-provider fallback: If your primary provider is down, route to a second provider offering a comparable model. This protects you from single-vendor outages, which have happened to every major provider.
Tiered-quality fallback: When a large flagship model is unavailable or too slow, fall back to a smaller, faster model. Responses may be slightly less capable, but the feature stays alive.
Region fallback: For cloud-hosted models, fail over to a different deployment region when one region is degraded.
Normalize your interface
Fallbacks only work cleanly if your code talks to models through a single abstraction layer. Wrap each provider behind a common interface that accepts a prompt and returns a normalized response. Then the fallback logic simply iterates through a prioritized list of adapters. This also makes testing and adding new models trivial.
Track which tier answered
Log the model and provider that actually produced each response. This is essential for debugging quality regressions and for measuring how often you're operating in a degraded state.
Step 4: Degrade Gracefully When Everything Fails
Eventually every model in your chain may be unavailable. Graceful degradation means returning something useful instead of a raw error.
Serve cached responses. For common or recently seen prompts, a cache can answer instantly and shield you from outages entirely. Semantic caching, which matches similar prompts, extends this further.
Return a partial or templated result. If an AI summary is unavailable, show the raw source content with a note that AI enhancement is temporarily unavailable.
Queue and defer. For non-urgent work, accept the request, return an acknowledgment, and process it later when capacity returns.
Fail honestly. When there is truly no fallback, return a clear, user-friendly message rather than a stack trace. Never leave the UI spinning indefinitely.
Step 5: Protect the System with Circuit Breakers
Retries and fallbacks help individual requests, but a circuit breaker protects the whole system. When a provider's error rate crosses a threshold, the breaker "opens" and short-circuits calls to that provider for a cooldown window — skipping straight to the fallback. This stops you from wasting time and money retrying a dead endpoint and gives the provider room to recover before you probe it again with a trickle of traffic.
Observability: You Can't Fix What You Can't See
Every layer of this pattern should emit telemetry. Track per-provider latency percentiles, retry counts, fallback activation rates, circuit-breaker state changes, and cache hit rates. Set alerts on fallback rate spikes — a sudden jump usually signals an upstream incident before your users start complaining.
Putting It Together
A resilient LLM request flow looks like this: apply a tight timeout, retry transient errors with backoff and jitter, fall through to alternative models via a normalized adapter layer, guard each provider with a circuit breaker, and degrade gracefully to caches or templated responses when all else fails — logging everything along the way.
None of these techniques are novel; they're battle-tested patterns from distributed systems engineering. What's new is applying them to the unpredictable latency and semantic quirks of LLM inference. Teams that internalize this discipline ship AI features that stay dependable in production, even when the models underneath them don't.
Ready to build real AI skills? Join the September 2026 cohort at Class For Jobs. Explore Advanced AI — a hands-on, live program to build and ship production AI applications, live and instructor-led with career support, resume help, and job-placement assistance.
Related reading
