Drift, Feedback Loops & Decision Debt

A decision system that worked at launch will not keep working on its own. Customers change, competitors move, data pipelines break, and the system’s own decisions reshape the data it learns from. The decay is usually quiet. Nobody notices until a quarterly review shows the programme’s value has halved.

This lesson is about seeing decay early and preventing the kinds that are self-inflicted.

Three kinds of drift

Data drift (covariate shift). The inputs change: new customer acquisition channels, a new product, a seasonal mix. The relationship between inputs and outcomes may still hold, but the model is now making predictions in regions it saw little of.

Prior shift (label shift). The base rate changes, as when fraud doubles during an attack or churn rises after a price increase. Rankings often stay good while probabilities become wrong. Decisions that compare probabilities with costs go wrong immediately. The fix is cheap if you catch it: recalibrate or apply the prior-shift correction from Lesson 4.

Concept drift. The relationship itself changes: an offer that used to persuade no longer does because a competitor launched a cheaper plan; fraudsters change tactics. The model is now wrong in ways no amount of recalibration fixes. It needs new labels and retraining, and possibly new features.

A monitoring stack for decisions

Monitor every layer, ordered from fastest to slowest signal:

The population stability index (PSI) is a common, simple drift score for inputs and scores:

import numpy as np

def psi(expected: np.ndarray, actual: np.ndarray, bins: int = 10) -> float:
    """Population stability index between a reference sample (e.g. training
    window) and a recent sample. Rule of thumb: < 0.1 stable, 0.1-0.25
    moderate shift, > 0.25 large shift worth investigating."""
    edges = np.unique(np.quantile(expected, np.linspace(0, 1, bins + 1)))
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(np.clip(actual, edges[0], edges[-1]), edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))

Fast signals don’t prove harm, but sudden changes in them are the best early warning you have, and they catch the most common real-world failure: a broken upstream pipeline. A feature that’s suddenly 40% null, a score distribution that jumps overnight, or an action mix that shifts on a Tuesday is almost always a bug, not a market change.

Outcome monitoring closes the loop: once labels arrive, compare predicted and realized rates per score bucket (calibration), and compare per action (are the offers still working?). And the holdout comparison from Lesson 19 is the final word on whether the system still adds value.

Feedback loops

The subtlest decay is self-inflicted: the decision policy shapes the data the next model learns from.

• Selective labels. A credit model sees repayment outcomes only for approved applicants. If it wrongly declines a group, it never sees evidence that the group would have repaid. Retrain after retrain, it can grow more conservative with nothing in the data to push back.
• Treatment shapes outcomes. A churn model trained on data where high-risk customers received offers learns that high-risk customers churn less than expected (because the offers worked), and starts scoring them lower, which stops the offers, which brings the churn back. The system oscillates.
• Performative effects. Customers adapt to the policy: they learn that visiting the cancel page triggers a discount, or fraudsters learn which transaction patterns get approved.
• Exploration collapse. A bandit or a greedy policy stops trying an action, so no new evidence about it arrives, so it never gets tried again.

The countermeasures are the ones built throughout this course:

• Keep a randomized holdout and exploration floor, so every action keeps getting some unconfounded data.
• Log propensities and treatment, and train on the right target: outcome under each action (Module 4), not the raw outcome mixed with whatever the policy did.
• For selective labels, explore at the margin: approve a small random share of borderline applicants, under explicit risk limits, to keep learning where the boundary really is.
• Watch for oscillation: track action rates and outcome rates by segment over successive model versions.

Decision debt

Decision systems accumulate debt in the logic around the model, not just in the model:

• Stale thresholds. A cutoff tuned for last year’s model, still hardcoded after three retrains shifted the score distribution. If thresholds aren’t derived from costs and calibrated probabilities (Lesson 5), every retrain silently changes decisions.
• Calibrators out of sync. A retrained model shipped with the old calibrator.
• Orphaned rules. Eligibility rules nobody owns, added for an incident three years ago.
• Permanent temporary overrides. “Exclude region X from offers for two weeks”, still in place 18 months later.
• Undocumented objectives. The trade-off weights in the objective were set in a meeting, and nobody remembers why.
• Unlogged decisions. Paths (often the fallback or manual ones) that don’t write to the decision log, so their decisions can’t be analyzed.

Pay it down on a schedule. A quarterly decision review with the owner works well:

1. Scorecard trends (Lesson 19): value, regret, overrides, guardrail trips.
2. Drift report: which inputs, priors or effects moved.
3. Rule and override audit: what fired, what never fired, what’s expired.
4. Assumption check: are cost estimates and trade-off weights still right? Rerun the sensitivity analysis from Lesson 5 with current numbers.
5. Decide: recalibrate, retrain, re-optimize, retire rules, or change the objective.

Monitoring and review keep the system correct. The final module makes it accountable and operable: fairness, explanations, audit trails, and the service architecture that runs all of this.