Comparing Gateways: LiteLLM, Portkey, TensorZero & Envoy AI Gateway

This course has built every hands-on exercise against LiteLLM since Module 3, and referenced Portkey, TensorZero, and Envoy AI Gateway conceptually since Lesson 2. This lesson puts all four side by side on the same task, configuring a fallback from a primary model to a self-hosted open-weight model, so the trade-offs are visible in real config rather than in a feature-comparison table alone.

The Same Policy, Four Configs

LiteLLM, the tool used throughout this course:

model_list:
  - model_name: production-chat
    litellm_params: {model: anthropic/claude-sonnet-4-5, api_key: os.environ/ANTHROPIC_API_KEY}
  - model_name: fallback-self-hosted
    litellm_params: {model: hosted_vllm/meta-llama/Llama-3.1-8B-Instruct, api_base: http://localhost:8000/v1, api_key: "none"}
litellm_settings:
  fallbacks: [{production-chat: [fallback-self-hosted]}]

Portkey, expressing the identical policy as one composable JSON object, its Gateway Config:

{
  "strategy": {"mode": "fallback"},
  "targets": [
    {"provider": "anthropic", "api_key": "sk-ant-...", "override_params": {"model": "claude-sonnet-4-5"}},
    {"provider": "openai", "api_base": "http://localhost:8000/v1", "override_params": {"model": "meta-llama/Llama-3.1-8B-Instruct"}}
  ]
}

TensorZero, in its tensorzero.toml, a Rust-based gateway built for sub-millisecond overhead at high QPS:

[models.production-chat]
routing = ["anthropic", "self_hosted_vllm"]

[models.production-chat.providers.anthropic]
type = "anthropic"
model_name = "claude-sonnet-4-5"

[models.production-chat.providers.self_hosted_vllm]
type = "vllm"
api_base = "http://localhost:8000/v1"
model_name = "meta-llama/Llama-3.1-8B-Instruct"

Envoy AI Gateway, Kubernetes-native, expressed as CRDs rather than a config file at all:

apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIGatewayRoute
metadata: {name: production-chat-route}
spec:
  parentRefs: [{name: envoy-ai-gateway, kind: Gateway, group: gateway.networking.k8s.io}]
  rules:
    - matches: [{headers: [{type: Exact, name: x-ai-eg-model, value: production-chat}]}]
      backendRefs:
        - {name: anthropic-backend, priority: 0}
        - {name: self-hosted-vllm-backend, priority: 1}

What Actually Differs

flowchart LR
    LiteLLM["LiteLLM\nsingle process,\nYAML config"] --> Fit1["Fastest start,\nbroadest provider list,\nthis course's tool"]
    Portkey["Portkey\nNode process,\ncomposable JSON config"] --> Fit2["Rich per-request policy,\n50+ built-in guardrails"]
    TensorZero["TensorZero\nRust binary,\nTOML config"] --> Fit3["Lowest overhead,\nbuilt-in eval/experimentation"]
    Envoy["Envoy AI Gateway\nKubernetes CRDs"] --> Fit4["Fits teams already\nrunning Envoy Gateway"]

    style LiteLLM fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style Portkey fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style TensorZero fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style Envoy fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style Fit1 fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style Fit2 fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style Fit3 fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style Fit4 fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px

All four implement the same six layers from Lesson 2, and all four can express the same fallback policy shown above, the real differentiator is not “can it do X” for most core features covered in this course, but deployment model fit and configuration ergonomics for a specific team. TensorZero’s Rust implementation genuinely earns a different overhead profile at very high QPS, a real technical differentiator, not just a stylistic one. Envoy AI Gateway’s CRD model is the right choice specifically when a platform team already operates Envoy Gateway and wants AI routing to be one more thing managed the same way as the rest of their traffic, not a separate paradigm. Portkey’s composable config is a strong fit when policy needs to vary dynamically per request rather than being fixed at deploy time. LiteLLM, this course’s tool, earns its place as the default starting point for its combination of the broadest provider coverage and the gentlest path from zero to a running, config-driven gateway.

With every architectural piece covered, routing, reliability, caching, governance, observability, and deployment, and four real gateways compared, the capstone in Lesson 28 puts all of it together in one project: designing and building a production multi-LLM gateway end to end.