Quality & Drift Monitoring Across Models

Lesson 22 instrumented every request for latency, tokens, and errors. None of that catches a specific, real production failure mode: a deployment that keeps returning perfectly successful 200 responses, at normal latency, while the actual quality of its answers quietly gets worse. This happens for real reasons, a provider updates the model version behind a stable-sounding name, a self-hosted deployment’s configuration drifts, and it is invisible to every signal Lesson 22 tracks. This lesson builds the layer that catches it.

LLM-as-Judge: Scoring Sampled Responses

Human review of every production response doesn’t scale. LLM-as-judge substitutes a separate scoring call, run against a sample of real traffic, using a strong model and an explicit rubric:

import json
from openai import OpenAI

judge_client = OpenAI(api_key="sk-...")  # a strong, stable model used consistently as the judge

JUDGE_PROMPT = """Rate the following AI assistant response on a 1-5 scale for:
- Correctness: is the information accurate?
- Relevance: does it actually address the user's request?
- Completeness: is anything important missing?

User request: {user_request}
Assistant response: {response}

Respond with strict JSON: {{"score": 1-5, "reasoning": "one sentence"}}"""

def judge_response(user_request: str, response_text: str) -> dict:
    result = judge_client.chat.completions.create(
        model="gpt-4o",              # a fixed, stable judge model, kept constant so scores are comparable over time
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(
            user_request=user_request, response=response_text,
        )}],
        temperature=0,
        response_format={"type": "json_object"},
    )
    return json.loads(result.choices[0].message.content)

judge_response(
    user_request="What's the refund window for a defective item?",
    response_text="Defective items can be returned within 30 days for a full refund.",
)
# -> {"score": 5, "reasoning": "Directly and accurately answers the question."}

Keeping the judge model itself fixed and stable is important: if the judge changes, score movements become impossible to attribute to the deployment being evaluated versus a change in how it’s being scored.

Sampling and Aggregating in the Gateway Pipeline

import random

JUDGE_SAMPLE_RATE = 0.05   # score 5% of production traffic, enough for a stable trend without judging every call

def maybe_score_response(deployment: str, user_request: str, response_text: str):
    if random.random() > JUDGE_SAMPLE_RATE:
        return
    result = judge_response(user_request, response_text)
    write_quality_score(deployment=deployment, score=result["score"], timestamp=now())

-- Rolling daily average quality score per deployment
SELECT
    deployment,
    date_trunc('day', timestamp) AS day,
    AVG(score) AS avg_score,
    COUNT(*) AS sample_count
FROM quality_scores
WHERE timestamp >= now() - interval '30 days'
GROUP BY deployment, day
ORDER BY deployment, day;

Detecting Drift

Drift is a sustained, meaningful decline in a deployment’s average score with no corresponding change on the gateway’s own side, exactly the signature that points outward, at something changing about the deployment itself:

def detect_drift(deployment: str, baseline_window_days=14, recent_window_days=3, threshold=0.5) -> bool:
    baseline_avg = get_avg_score(deployment, days_back=baseline_window_days, days_back_end=recent_window_days)
    recent_avg = get_avg_score(deployment, days_back=recent_window_days, days_back_end=0)

    if baseline_avg - recent_avg > threshold:
        alert(f"Quality drift detected for {deployment}: "
              f"baseline {baseline_avg:.2f} -> recent {recent_avg:.2f}")
        return True
    return False

flowchart LR
    Prod["Production traffic"] --> Sample["Sample ~5%\n(JUDGE_SAMPLE_RATE)"]
    Sample --> Judge["LLM-as-judge scoring\n(fixed judge model)"]
    Judge --> Store["Rolling per-deployment\nscore history"]
    Store --> Compare["Compare recent window\nvs baseline window"]
    Compare -->|"meaningful decline"| Alert["Drift alert:\ninvestigate deployment"]
    Compare -->|"stable"| OK["No action"]

    style Prod fill:#EEF0F7,stroke:#6366F1,color:#0F172A,rx:8px,ry:8px
    style Judge fill:#4338CA,stroke:#4338CA,color:#ffffff,rx:8px,ry:8px
    style Alert fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px
    style OK fill:#F7F8FC,stroke:#0D9488,color:#0F172A,rx:8px,ry:8px

A drift alert on a deployment is information the routing engine (Module 4) should be able to act on, exactly as it would react to a latency or availability signal: temporarily deprioritizing a drifting deployment, or triggering a manual review of whether it still belongs in a fallback chain (Lesson 14) at its previous tier. Lesson 24 closes Module 8 with the complementary practice for a planned change rather than an unplanned drift: safely rolling out a model upgrade using A/B testing and shadow traffic.