Latency- and Cost-Aware Dynamic Routing

Lesson 10 ended on static routing’s blind spot: a deployment that is technically up but running slow gets no special treatment from a fixed priority order. This lesson closes that gap with two dynamic strategies LiteLLM’s router already implements, one reacting to latency, one to cost, both continuously, not just at config-write time.

Latency-Based Routing

from litellm import Router

model_list = [
    {
        "model_name": "production-chat",
        "litellm_params": {"model": "openai/gpt-4o", "api_key": "sk-openai-..."},
    },
    {
        "model_name": "production-chat",
        "litellm_params": {
            "model": "hosted_vllm/meta-llama/Llama-3.1-8B-Instruct",
            "api_base": "http://localhost:8000/v1",
            "api_key": "none",
        },
    },
]

router = Router(
    model_list=model_list,
    routing_strategy="latency-based-routing",
    enable_pre_call_checks=True,   # skip deployments already known to be failing before even attempting a call
)

response = router.completion(
    model="production-chat",
    messages=[{"role": "user", "content": "Draft a one-line status update."}],
)

The router maintains a rolling measurement of each deployment’s recent response time and biases new requests toward whichever deployment has actually been responding faster lately, no config change required when conditions shift. Concretely: if the cloud provider’s gpt-4o deployment starts responding in 6 seconds instead of its usual 1.5 (a real, observed pattern during provider-side load spikes), the router shifts more traffic to the self-hosted deployment automatically, and shifts back once latency recovers. This is precisely the scenario static routing cannot handle: the cloud deployment never returns an error, it is just slow, so an order-based strategy has no signal to react to at all.

Cost-Based Routing

router = Router(
    model_list=model_list,
    routing_strategy="cost-based-routing",
)

Where latency-based routing reacts to a measured runtime signal, cost-based routing optimizes against a static but real one: each provider’s published per-token pricing (which LiteLLM tracks internally per model). Given deployments of comparable quality for a task, cost-based-routing prefers the cheaper one directly, expressing the same intent a team might otherwise encode by hand in a static order field, but derived from actual pricing instead of a number an engineer chose and has to remember to update if prices change.

flowchart LR
    Req["Incoming request"] --> Router["Router\n(latency-based OR cost-based)"]
    Router -->|"tracks rolling p50/p99\nresponse time"| Latency["Latency signal"]
    Router -->|"tracks $/million tokens\nper deployment"| Cost["Cost signal"]
    Latency --> Decision["Pick deployment"]
    Cost --> Decision
    Decision --> Fast["Currently-fastest\nor currently-cheapest\ndeployment"]

    style Router fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style Decision fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style Fast fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px

The Trade-Off: Predictability for Responsiveness

This is the direct counterpart to Lesson 10’s closing point. Static routing’s value was full predictability from the config file alone; dynamic routing gives that up in exchange for reacting to conditions no config could anticipate. Concretely, that means a request that gets routed to the self-hosted deployment during a latency spike, and to gpt-4o five minutes later once conditions normalize, is not something you can determine by reading config.yaml, you need to look at the actual latency measurements the router was tracking at that moment, which is exactly why Module 8’s observability work (unified tracing, per-request cost and latency logging) is not optional once dynamic routing is in play: it is how you reconstruct, after the fact, why a specific request went where it went.

In practice, most production gateways run a hybrid: a static priority order as the coarse-grained shape of the policy (which deployments are even eligible, and roughly which should be preferred), with a dynamic strategy operating within a tier to make the fine-grained call among comparable options. Lesson 12 adds a third, orthogonal axis to this decision: routing not by live performance signals but by classifying what kind of task the prompt itself represents.