A/B Testing & Shadow Traffic for Model Upgrades

Lesson 23 detected drift in a deployment already in production. This lesson covers the other side of change: deliberately introducing a new candidate deployment, a newer model version, a different provider, a self-hosted replacement, without risking production quality while you find out whether it’s actually as good as it looks on a benchmark. Two techniques, used in sequence, get this done: shadow traffic first, then a live A/B split.

Shadow Traffic: Zero Risk to Real Users

A shadow call sends a copy of a real production request to the candidate deployment purely for comparison, the user’s actual response always comes from the trusted, incumbent deployment:

import asyncio

async def handle_request_with_shadow(gateway_client, primary_model: str, candidate_model: str, messages: list[dict]):
    # The real, user-facing call: this response is what the caller receives.
    primary_response = await gateway_client.chat.completions.create(
        model=primary_model, messages=messages,
    )

    # Fire the shadow call concurrently, but never await it in the response
    # path and never let its result (or a failure in it) affect what the
    # user gets back. A shadow call failing should be invisible to the caller.
    asyncio.create_task(
        run_shadow_comparison(candidate_model, messages, primary_response)
    )

    return primary_response  # user only ever sees this

async def run_shadow_comparison(candidate_model, messages, primary_response):
    try:
        candidate_response = await gateway_client.chat.completions.create(
            model=candidate_model, messages=messages,
        )
        # Reuse Lesson 23's judge to score both, or compare them directly
        judge_result = judge_response(
            user_request=messages[-1]["content"],
            response_text=candidate_response.choices[0].message.content,
        )
        log_shadow_comparison(
            candidate_model=candidate_model,
            candidate_score=judge_result["score"],
            candidate_latency=candidate_response.response_ms,
        )
    except Exception as e:
        log_shadow_failure(candidate_model=candidate_model, error=str(e))

Because asyncio.create_task fires the shadow call without the primary response path ever waiting on or depending on its outcome, a candidate that crashes, times out, or returns malformed output has zero effect on any real user, that failure is exactly the information this stage exists to surface safely, before it could ever reach a live caller.

Live A/B Split: Weighted Traffic

Once shadow results look good, a live A/B split gives the candidate real (but limited and controlled) production exposure, using the same load-balancing weight mechanism from Lesson 9:

model_list:
  - model_name: production-chat
    litellm_params:
      model: anthropic/claude-sonnet-4-5      # incumbent
      api_key: os.environ/ANTHROPIC_API_KEY
      weight: 0.9                              # ~90% of live traffic

  - model_name: production-chat
    litellm_params:
      model: anthropic/claude-opus-4-6         # candidate: a newer model version
      api_key: os.environ/ANTHROPIC_API_KEY
      weight: 0.1                              # ~10% of live traffic, real users, real answers

At this stage, unlike shadow traffic, some real users are genuinely receiving the candidate’s actual answers, which is why it only happens after shadow traffic has already ruled out the most obvious failure modes. Lesson 22’s tracing and Lesson 23’s quality scoring both need gen_ai.request.model/served_by_model broken out per deployment here, so the candidate’s real-traffic latency and quality can be compared against the incumbent’s directly, not just against its own shadow-traffic numbers.

The Two-Stage Rollout

flowchart LR
    Candidate["New candidate deployment"] --> Shadow["Stage 1: Shadow traffic\nzero user-facing risk,\nreal request patterns"]
    Shadow -->|"failure modes found"| Fix["Fix candidate,\nretry shadow stage"]
    Shadow -->|"looks good"| AB["Stage 2: Live A/B split\n(weight: 0.1 or similar)"]
    AB -->|"quality/latency regression\n(Lesson 23 scoring)"| Rollback["Roll back to weight: 0"]
    AB -->|"comparable or better"| FullRollout["Increase weight toward 1.0,\neventually promote to primary"]

    style Candidate fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style Shadow fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style AB fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style FullRollout fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style Rollback fill:#F7F8FC,stroke:#94A3B8,color:#0F172A,rx:8px,ry:8px

Shadow traffic is not a substitute for a live A/B split, it can validate that a candidate doesn’t crash or produce obviously broken output, but it cannot fully validate real-world quality perception, since the judge scoring in Lesson 23 is itself an imperfect proxy for what real users actually think of an answer. The two stages are complementary: shadow traffic derisks the “does this even work” question before any user is exposed, and a live A/B split answers the “is this actually better” question with real signal, at a scale small enough to roll back cheaply if it isn’t. With Module 8’s observability and evaluation toolkit complete, tracing, drift detection, and safe rollout, Module 9 turns to running all of this in production infrastructure: Docker, Kubernetes, and multi-region deployment.