Fallback Chains & Graceful Degradation

Lesson 9 configured a basic two-tier fallback; Lesson 13 built the retry, timeout, and circuit-breaker mechanics that decide when a fallback should even engage. This lesson designs the fallback chain itself as a real production artifact: how many tiers, what triggers each transition, and, critically, what “success” even means when a fallback several tiers down answers with real but lower quality than the primary would have.

A Real Multi-Tier Chain

model_list:
  - model_name: production-chat
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: tier2-openai
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: tier3-self-hosted
    litellm_params:
      model: hosted_vllm/meta-llama/Llama-3.1-8B-Instruct
      api_base: http://localhost:8000/v1
      api_key: "none"

litellm_settings:
  fallbacks:
    - production-chat: ["tier2-openai", "tier3-self-hosted"]
  context_window_fallbacks:
    - production-chat: ["tier2-openai"]     # separate trigger: input too large for the primary's context limit
  num_retries: 2
  allowed_fails: 3                          # consecutive failures before this deployment is treated as down

Two distinct triggers are configured here, deliberately kept separate. fallbacks handles availability: the primary is down, erroring, or circuit-broken, so try the next tier. context_window_fallbacks handles a structurally different problem: the input is simply too large for the primary model’s context window, which has nothing to do with whether that model is healthy, a long document that needs summarizing should route to a larger-context model immediately, not after exhausting retries against a model that was never going to fit it regardless of health.

Quality-Aware Chain Design

flowchart TD
    T1["Tier 1: claude-sonnet-4-5\nprimary, highest quality"] -->|"unavailable"| T2["Tier 2: gpt-4o\ncomparable quality, different provider"]
    T2 -->|"unavailable"| T3["Tier 3: self-hosted Llama-3.1-8B\nlower quality, but independent\nof any cloud provider's uptime"]
    T3 -->|"unavailable"| Degrade["Graceful degradation:\ncached default / apology response"]

    style T1 fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style T2 fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style T3 fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style Degrade fill:#F7F8FC,stroke:#94A3B8,color:#0F172A,rx:8px,ry:8px

Notice tier 1 and tier 2 here are chosen to be roughly comparable in quality, both frontier-class models from different providers, so a fallback between them is close to invisible to the end user. Tier 3 is a real quality step down, a much smaller self-hosted model, included anyway because its independence from any cloud provider’s infrastructure makes it worth the quality trade during an outage severe enough to take out two major providers at once. A response that came from tier 3 is a technical success (a 200, a completed answer) but not necessarily an equivalent one, which is why it is worth tagging in the response metadata:

def call_with_tier_tracking(gateway_client, **kwargs):
    response = gateway_client.chat.completions.create(**kwargs)
    # LiteLLM includes which deployment actually served the request in
    # response headers/metadata; surface it so downstream code (and
    # dashboards, Module 8) can distinguish a tier-1 answer from a
    # degraded tier-3 one, rather than treating every 200 as equivalent.
    served_by = response.model  # the real model string that answered, not the alias requested
    if served_by != "claude-sonnet-4-5":
        log_degraded_response(requested="production-chat", served_by=served_by)
    return response

When Every Tier Fails

A chain needs an explicit last resort for the case where every configured deployment, cloud and self-hosted, is unavailable, a genuinely rare but real scenario (a broad network partition, a misconfigured shared credential). Silently returning a 500 to the end user is rarely the best option:

FALLBACK_RESPONSES = {
    "support_chat": "We're experiencing a temporary issue and can't generate a live response right now. Your message has been logged and a team member will follow up.",
}

def handle_total_failure(use_case: str) -> str:
    # A graceful degradation floor: a canned, honest response beats a raw
    # error surfaced to an end user, and it still resolves the request
    # instead of leaving a client hanging on a timeout.
    return FALLBACK_RESPONSES.get(use_case, "This service is temporarily unavailable. Please try again shortly.")

Fallbacks Are Not a Silent Safety Net

The most important operational rule for a fallback chain: every time it engages, it should be observable, ideally alertable, not just logged and forgotten. A team running on tier 3 for six hours because tier 1’s API key silently expired is a real incident that a fully silent fallback chain will hide from everyone until a user complains about answer quality. Lesson 15 covers the complementary mechanism, rate limiting and load shedding, for protecting the gateway (and its own callers) when the problem is not a failing provider but too much legitimate traffic arriving at once.