Module 3 built a fallback chain, production-chat failing over to a self-hosted Llama deployment. That fallback was implicitly a two-tier priority order. This lesson makes that ordering explicit and configurable across more than two tiers, and treats it as what it actually is: the simplest correct routing strategy, and the one every team should start with before reaching for anything that reacts to live conditions.
Configuring Explicit Priority
model_list:
- model_name: production-chat
litellm_params:
model: azure/gpt-4o-primary
api_base: https://team-primary.openai.azure.com/
api_key: os.environ/AZURE_API_KEY_PRIMARY
order: 1 # tried first
- model_name: production-chat
litellm_params:
model: azure/gpt-4o-secondary
api_base: https://team-secondary.openai.azure.com/
api_key: os.environ/AZURE_API_KEY_SECONDARY
order: 2 # tried only if all order=1 deployments are unavailable
- model_name: production-chat
litellm_params:
model: hosted_vllm/meta-llama/Llama-3.1-8B-Instruct
api_base: http://localhost:8000/v1
api_key: "none"
order: 3 # last resort: self-hosted, independent of any cloud provider
litellm_settings:
num_retries: 2order is a strict preference ranking, not a load-balancing weight: the router always prefers the lowest order value available, and only descends to the next tier when every deployment at the current tier is unavailable. This differs from simple-shuffle (Lesson 9), which treats same-tier deployments as an undifferentiated pool to spread across; here, two order: 1 deployments (say, two Azure regions of the same model) would still be load-balanced between each other, while an order: 2 deployment only sees traffic once both order: 1 options are gone.
Rate Limits Get Special Treatment
A 429 (rate limited) response is architecturally different from a 500 (server error): it is a known, temporary, deployment-specific condition, not a general failure. LiteLLM’s router reflects this: a rate-limited deployment is placed on a cooldown timer, and while it is cooling down, the router can move straight to the next order tier rather than retrying the same deployment or walking through a slower generic fallback path.
flowchart TD
Req["Request: production-chat"] --> O1{"order=1 deployments\navailable?"}
O1 -->|"Yes, healthy"| Serve1["Serve from order=1\n(load-balanced if multiple)"]
O1 -->|"429: rate limited"| Cooldown["Place on cooldown timer"]
Cooldown --> O2{"order=2 available?"}
O1 -->|"500: hard failure"| O2
O2 -->|Yes| Serve2["Serve from order=2"]
O2 -->|No| O3["order=3: self-hosted last resort"]
style Req fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
style Serve1 fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
style Serve2 fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
style O3 fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8pxWhy Start Here
Static routing’s entire value proposition is predictability. Anyone reviewing config.yaml can state exactly what will happen in any failure scenario without running anything: “if the primary Azure region rate-limits us, traffic moves to the secondary region; if both Azure deployments are down, we serve from the self-hosted fallback.” That property, being fully auditable from a config diff, is worth a lot during an incident, when an on-call engineer needs to reason about gateway behavior under pressure, and it is worth a lot during code review, when a teammate needs to evaluate whether a proposed priority order actually makes sense for the business (should the cheaper self-hosted model really be order: 3, or should cost make it order: 1 for non-critical traffic?).
Static routing’s limitation is exactly the flip side of that predictability: it cannot react to anything the config didn’t anticipate. If the primary deployment is technically “up” but running at 8-second latency due to load, static routing has no mechanism to notice and route around it, order-based routing only reacts to hard failures and rate limits, not degraded performance. That gap is precisely what Lesson 11’s latency- and cost-aware dynamic routing exists to close.