Deploying the Gateway: Docker & Kubernetes

Every lesson so far ran the LiteLLM proxy as a local process. This lesson deploys it for real: a proper Dockerfile, the Redis dependency load-balancing and rate limiting have depended on since Lessons 9 and 15, and a Kubernetes deployment sized around how a gateway actually behaves under load, which is different from a typical stateless web service.

Containerizing the Gateway

FROM python:3.12-slim

WORKDIR /app
RUN pip install --no-cache-dir 'litellm[proxy]' redis

# config.yaml is NOT baked into the image: it's mounted at runtime (via
# ConfigMap in Kubernetes, below) so a routing change is a config update,
# not an image rebuild and redeploy.
EXPOSE 4000

CMD ["litellm", "--config", "/etc/litellm/config.yaml", "--port", "4000", "--num_workers", "4"]

# docker-compose.yml: local multi-container setup, gateway + the Redis
# dependency used since Lesson 9 (usage-based routing) and Lesson 15
# (distributed rate limiting)
services:
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]

  gateway:
    build: .
    ports: ["4000:4000"]
    volumes:
      - ./config.yaml:/etc/litellm/config.yaml:ro
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - REDIS_HOST=redis
    depends_on: [redis]

Secrets flow in as environment variables from the deployment environment, not baked into the image, exactly the master-key/virtual-key separation from Module 3 taken to its logical conclusion: an image with a real provider key baked in is extractable by anyone who can pull it, which defeats the whole point of keeping raw provider credentials out of anything application code (or, here, a distributable artifact) can touch.

Kubernetes: Deployment, Service, Secrets

apiVersion: v1
kind: ConfigMap
metadata:
  name: litellm-config
data:
  config.yaml: |
    model_list:
      - model_name: production-chat
        litellm_params:
          model: anthropic/claude-sonnet-4-5
          api_key: os.environ/ANTHROPIC_API_KEY
    router_settings:
      redis_host: redis-service
      redis_port: 6379
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-gateway
spec:
  replicas: 3                          # multiple replicas: Redis (below) keeps their routing/rate-limit state consistent
  selector:
    matchLabels: {app: ai-gateway}
  template:
    metadata:
      labels: {app: ai-gateway}
    spec:
      containers:
        - name: ai-gateway
          image: ai-gateway:latest
          ports: [{containerPort: 4000}]
          envFrom:
            - secretRef: {name: ai-gateway-provider-keys}   # real provider keys, from a real secrets manager
          volumeMounts:
            - {name: config, mountPath: /etc/litellm}
          readinessProbe:
            httpGet: {path: /health/readiness, port: 4000}
            initialDelaySeconds: 5
          livenessProbe:
            httpGet: {path: /health/liveliness, port: 4000}
            initialDelaySeconds: 10
      volumes:
        - name: config
          configMap: {name: litellm-config}
---
apiVersion: v1
kind: Service
metadata:
  name: ai-gateway-service
spec:
  selector: {app: ai-gateway}
  ports: [{port: 80, targetPort: 4000}]

Wiring readinessProbe/livenessProbe to the proxy’s own health endpoints matters specifically for a gateway: a replica that can accept TCP connections but whose upstream provider connectivity is broken should be pulled from rotation by the readiness check, not left serving requests it can’t actually fulfill.

Sizing the Autoscaler for I/O-Bound Load

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ai-gateway-hpa
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: ai-gateway}
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric: {name: in_flight_requests}   # custom metric, not raw CPU
        target: {type: AverageValue, averageValue: "50"}

flowchart TD
    LB["Kubernetes Service"] --> P1["Gateway pod 1"]
    LB --> P2["Gateway pod 2"]
    LB --> P3["Gateway pod 3"]
    P1 --> Redis[("Shared Redis\nrouting + rate-limit state")]
    P2 --> Redis
    P3 --> Redis
    HPA["HPA: in_flight_requests"] -.->|scales| P1
    HPA -.->|scales| P2
    HPA -.->|scales| P3

    style LB fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style Redis fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style HPA fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px

Raw CPU is a weak autoscaling signal here because a gateway is fundamentally I/O-bound, most of the time handling a request is spent waiting on a provider’s network response, not consuming CPU, so CPU usage can stay deceptively low right up until the gateway is genuinely saturated on concurrency. A metric that reflects real load, in-flight request count (shown above) or p99 latency, scales the deployment based on what’s actually happening rather than a proxy signal that lags behind it. With the gateway containerized and deployed, Lesson 26 extends this to multiple regions and the failure scenarios a single-region deployment, however well autoscaled, still can’t survive.