LLM App Patterns: Routing, Chaining, and Fallbacks
Building a demo with a single prompt is easy. Building an LLM application that stays reliable under real traffic, edge cases, and cost constraints is a different discipline. The teams that ship durable AI products don't rely on one giant prompt — they compose smaller, predictable steps using well-understood LLM design patterns.
This is a practical catalog of three foundational patterns — routing, chaining, and fallbacks — plus the operational concerns that make them production-ready. Treat these as reusable building blocks you can mix and match across projects.
Why You Need LLM Design Patterns
A raw LLM call is stochastic, occasionally wrong, and priced per token. When you wrap that call in structure, you gain control over three things that matter most in production: accuracy, cost, and latency. Design patterns give you a shared vocabulary for making those tradeoffs deliberately instead of accidentally.
The patterns below assume you already have basic instrumentation in place: request logging, token counting, and some form of evaluation. Without measurement, you're guessing.
Pattern 1: Routing
Routing sends each incoming request to the most appropriate handler. Instead of forcing one model or one prompt to do everything, you classify the request first, then dispatch.
When to use it
Use routing when your traffic contains distinct task types — for example, a support assistant that handles billing questions, technical troubleshooting, and general FAQ. Each category benefits from a different prompt, model tier, or tool set.
How it works
A lightweight classifier — often a small, fast model or even a rules-based check — inspects the input and returns a route label. The application then forwards the request to the matching pipeline.
- Model-based routing: A cheap model like a small instruction-tuned LLM decides the category. Ask it to return a strict enum value so parsing stays trivial.
- Cost-based routing: Send simple queries to a smaller, cheaper model and escalate only complex ones to a frontier model. This alone can cut inference spend dramatically.
- Semantic routing: Embed the query and match it against category exemplars using vector similarity — useful when categories are fuzzy.
Implementation tips
Keep the router's output constrained. Request a single token or a JSON field with a fixed set of allowed values, and validate it. Always define a default route for inputs the classifier can't confidently label, so nothing falls through silently.
Pattern 2: Chaining
Chaining decomposes a complex task into a sequence of smaller steps, where each step's output feeds the next. Rather than asking one prompt to research, summarize, and format all at once, you split the work.
When to use it
Reach for chaining when a task has natural sub-steps, when a single prompt produces inconsistent results, or when intermediate outputs need validation before you continue. Document generation, data extraction with verification, and multi-stage reasoning are common cases.
Common chain shapes
- Sequential chain: Step A extracts entities, step B enriches them, step C writes the final response. Each step has a focused prompt and is easier to test in isolation.
- Prompt-then-parse: One call generates content, a second call (or a deterministic function) validates and reformats it. Splitting generation from formatting improves reliability.
- Map-reduce: For long inputs that exceed practical context limits, process chunks in parallel (map), then combine the partial results (reduce). Useful for summarizing large document sets.
Implementation tips
Define a clear data contract between steps — ideally structured output such as JSON with a schema. Validate at each boundary so a bad intermediate result fails fast instead of corrupting everything downstream. Log intermediate outputs; when a chain misbehaves, you'll want to see exactly which step went wrong. Be mindful that chaining multiplies latency and cost, so only add steps that earn their keep.
Pattern 3: Fallbacks
Fallbacks keep your application working when the primary path fails. Models return malformed output, providers have outages, rate limits trigger, and some prompts simply produce low-confidence answers. A fallback strategy handles these gracefully instead of surfacing an error to the user.
Types of fallbacks
- Provider fallback: If your primary model endpoint times out or returns an error, retry against a secondary provider or a different model. This protects you against single-vendor outages.
- Model tier fallback: If a smaller model returns a low-confidence or unparseable result, escalate to a stronger model for that specific request.
- Format fallback: If structured output fails validation, retry with a stricter prompt, or attempt a repair pass that fixes the malformed JSON.
- Graceful degradation: When all model options fail, return a safe canned response, a cached answer, or a handoff to a human — never an unhandled exception.
Implementation tips
Wrap calls with sensible timeouts and bounded retries using exponential backoff. Distinguish between retryable errors (rate limits, transient network issues) and non-retryable ones (invalid requests) so you don't waste calls. Track how often each fallback fires — a rising fallback rate is an early warning that something upstream is degrading.
Combining the Patterns
These patterns are most powerful together. A realistic production flow might look like this:
- A router classifies an incoming request and picks the right pipeline.
- The chosen pipeline runs a chain — extract, reason, format — with validation between steps.
- Each model call is wrapped in fallback logic for provider outages and output-format failures.
Layering them this way isolates concerns: routing controls cost and specialization, chaining controls accuracy and complexity, and fallbacks control reliability. You can improve each independently without rewriting the whole system.
Operational Guardrails That Make Patterns Work
Patterns only pay off when supported by good engineering hygiene. Keep these in place across every pattern:
- Evaluation harness: Maintain a test set of representative inputs and expected behaviors. Run it whenever you change a prompt, model, or route.
- Observability: Log inputs, outputs, latency, token counts, and which route or fallback fired. You cannot debug what you cannot see.
- Cost budgets: Set per-request and per-day spending limits, and alert when routing sends too much traffic to expensive models.
- Caching: Cache deterministic or repeated calls to cut both latency and cost, especially for router classifications and common queries.
- Idempotency and timeouts: Make retries safe and bound every external call so a slow provider can't hang your whole request.
Choosing the Right Pattern
Start simple. A single well-crafted prompt is often enough for a first version. Add patterns only when a specific problem appears: introduce routing when traffic diversifies and costs climb, add chaining when one prompt can't hold accuracy across sub-tasks, and build fallbacks the moment you take real user traffic. Each pattern adds complexity, so let observed pain — not speculation — drive adoption.
Mastering these three patterns gives you a dependable toolkit for turning fragile prototypes into resilient AI systems. They're vendor-neutral, framework-agnostic, and durable across the fast-moving model landscape of 2026.
Ready to build real AI skills? Join the September 2026 cohort at Class For Jobs. Explore Advanced AI — a hands-on, live program to build and ship production AI applications, live and instructor-led with career support, resume help, and job-placement assistance.
Related reading
