Capstone: A Production Multi-LLM Gateway End-to-End

Every lesson in this course built one layer of a gateway in isolation. This capstone combines all of them into a single, coherent system for a realistic scenario: Meridian, a mid-size logistics company running a customer support bot, an internal engineering copilot, and a fraud-detection service, all routing through one gateway.

The Requirements

• Support bot (customer-facing): needs the highest available quality, a $2,000/month budget, and must survive a full outage of the primary cloud region without going down entirely.
• Internal engineering copilot: needs frontier-model access for complex reasoning, a $500/month budget, and audit logging (some engineers work with regulated customer data in prompts).
• Contractor access (a rotating set of short-term contractors): restricted to a self-hosted open-weight model only, capped at $20/month, no exceptions.
• Company-wide: PII must never reach an external provider unredacted; every team’s spend must be attributable in a monthly report; quality must be monitored so a silent provider-side model change doesn’t go unnoticed.

Assembling the Config

model_list:
  # Support bot: primary + fallback, prioritizing availability (Lesson 14)
  - model_name: support-chat
    litellm_params: {model: anthropic/claude-sonnet-4-5, api_key: os.environ/ANTHROPIC_API_KEY, order: 1}
  - model_name: support-chat
    litellm_params: {model: openai/gpt-4o, api_key: os.environ/OPENAI_API_KEY, order: 2}
  - model_name: support-chat
    litellm_params: {model: hosted_vllm/meta-llama/Llama-3.1-8B-Instruct, api_base: http://vllm-internal:8000/v1, api_key: "none", order: 3}

  # Internal copilot: frontier access, latency-aware between two comparable options (Lesson 11)
  - model_name: internal-copilot
    litellm_params: {model: anthropic/claude-sonnet-4-5, api_key: os.environ/ANTHROPIC_API_KEY}
  - model_name: internal-copilot
    litellm_params: {model: openai/gpt-4o, api_key: os.environ/OPENAI_API_KEY}

  # Contractor pool: self-hosted only, no cloud provider access at all
  - model_name: contractor-safe
    litellm_params: {model: hosted_vllm/meta-llama/Llama-3.1-8B-Instruct, api_base: http://vllm-internal:8000/v1, api_key: "none"}

router_settings:
  routing_strategy: latency-based-routing   # Lesson 11: internal-copilot picks the faster of two comparable models
  redis_host: redis-service                  # Lesson 25: shared state across replicas

litellm_settings:
  fallbacks:
    - support-chat: []                       # already expressed via order above; retained for clarity
  num_retries: 2
  cache: true                                # Lesson 16: exact-match caching on by default
  cache_params: {type: redis, similarity_threshold: 0.95}  # Lesson 16: semantic cache threshold

general_settings:
  master_key: os.environ/GATEWAY_MASTER_KEY

Wiring Governance and Observability Around It

# Team/key setup (Lesson 19), run once at provisioning time
TEAMS = {
    "support-bot":        {"models": ["support-chat"], "max_budget": 2000},
    "internal-eng":       {"models": ["internal-copilot"], "max_budget": 500},
    "contractors":        {"models": ["contractor-safe"], "max_budget": 20},
}

# Every request path: PII redaction (Lesson 20) before routing, audit log
# (Lesson 21) on every outcome, cost logging (Lesson 18) at egress
def handle_gateway_request(request, key_metadata):
    request = gateway_pre_call_hook(request)          # PII redaction, Lesson 20
    try:
        check_budget_before_call(key_metadata["key_id"], estimate_cost(request))  # Lesson 18
    except PermissionError as e:
        write_audit_entry(AuditLogEntry(..., outcome="rejected_budget", detail=str(e)))
        raise

    response = router.completion(**request)            # Module 4 routing + Module 5 reliability

    cost = compute_cost_from_usage(response)            # Lesson 18
    log_request_cost(key_metadata, response.model, cost, now())
    write_audit_entry(AuditLogEntry(..., outcome="success", cost_usd=cost))

    maybe_score_response(response.model, request, response)  # Lesson 23: sampled quality monitoring
    return response

Deployment and Resilience

flowchart TD
    subgraph USEast["us-east (primary region)"]
        GWEast["Gateway replicas\n(K8s Deployment, Lesson 25)"]
    end
    subgraph EUWest["eu-west (failover region)"]
        GWWest["Gateway replicas"]
    end
    DNS["Geo DNS / health-check failover\n(Lesson 26)"]
    DNS --> GWEast
    DNS -.->|"failover if us-east unreachable"| GWWest

    SupportBot["Support bot"] --> DNS
    Copilot["Internal copilot"] --> DNS
    Contractors["Contractor tools"] --> DNS

    style DNS fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style GWEast fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style GWWest fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px

The support-chat model’s three-tier order (Lesson 14) handles a single provider or deployment failing; the two-region topology with geo-DNS failover (Lesson 26) handles the harder case of the whole primary region’s gateway becoming unreachable, a scenario the in-process fallback chain alone has no visibility into. Both are present because the requirement, “must survive a full regional outage,” specifically demanded the layer that fallback chains alone don’t cover.

What’s Deliberately Out of Scope for Launch

A real rollout distinguishes what a launch cannot safely operate without from what improves the system over time. This capstone’s first version ships with authorization, budgets, PII redaction, basic fallback, and audit logging, the mechanisms whose absence would be a safety or correctness gap. It explicitly defers a full LLM-as-judge drift-monitoring pipeline (Lesson 23) and a live A/B rollout process (Lesson 24) to a fast-follow phase: their absence at launch is a cost-optimization and quality-improvement gap, not a reason traffic can’t safely flow through the gateway on day one. Scoping a launch this way, and being able to articulate why each deferred piece is safe to defer, is itself a skill this course was building toward as much as any single lesson’s mechanism.

That is the complete arc: from Lesson 1’s four production failures that motivated a gateway at all, through the request lifecycle and schema that make routing possible, the LiteLLM build, the routing strategies, the reliability engineering, the cost and governance controls, the observability layer, and finally deployment at scale, culminating in one system that has to satisfy all of it at once, exactly as a real platform team’s gateway does.