Experimentation for Decision Policies

Lesson 15 used off-policy evaluation to screen candidate policies. The survivors need an online test before they replace the current policy: real customers, real outcomes, randomized assignment. Policy tests look like ordinary A/B tests, but decision systems add complications that break naive designs.

Designing the test

A policy test compares two decision policies (current vs candidate), not two versions of a page. Write down before starting:

• Randomization unit: the customer, account, store or region that is assigned to a policy for the whole test. Use the unit the decision is made for, and keep assignment sticky so a customer doesn’t flip between policies.
• Primary metric: the decision objective (net margin over the horizon, including costs). One metric, decided in advance.
• Guardrail metrics: things that must not get worse beyond a tolerance, such as complaint rate, unsubscribe rate, approval rate for a protected group, latency.
• Minimum detectable effect and sample size: the smallest improvement worth detecting, and the number of units needed to detect it with reasonable power.
• Duration: long enough for outcomes to arrive and to cover business cycles (at least full weeks, often a full month for monthly decisions).

Policy changes often have small effects on noisy money metrics. A power calculation done honestly sometimes shows the test would need a year. Better to know that before starting, and then reach for the variance reduction below, a proxy metric, or a larger effect.

Interference: when arms affect each other

Standard A/B analysis assumes one unit’s treatment doesn’t affect another’s outcome. Decision systems violate this in three common ways.

Shared budgets and capacity. If both policies draw from one offer budget or one review team, an aggressive policy starves the other. Fix: split resources in proportion to traffic. With a 50/50 split, each arm gets 50% of the budget, 50% of the call capacity, its own budget price λ. Each arm then behaves as it would at full rollout, scaled down.

Marketplaces and shared supply. Dispatch, pricing and matching policies in marketplaces affect each other through the shared pool of drivers, inventory or advertisers. Randomizing individual orders makes the treatment “steal” supply from control and exaggerates its effect. Fix: switchback experiments, which randomize whole blocks of time within a region (for example, 1-hour blocks alternate between policies), or cluster randomization by city or store.

Network effects and learning. Customers talk, share referral codes, and learn that waiting brings a discount. Effects that spread through these channels are hard to contain. Cluster randomization (by region or social cluster) helps; long-running holdouts reveal effects that only show up over months.

Variance reduction: CUPED

Money metrics are noisy, so tests take long. CUPED (Controlled-experiment Using Pre-Experiment Data, Deng et al., 2013) reduces variance using a covariate measured before the test, typically the same metric in a pre-period:

import numpy as np

def cuped_adjust(y, x_pre):
    """y: outcome during the test; x_pre: same metric in the pre-period.
    Pool both arms to estimate theta, then adjust every unit."""
    theta = np.cov(y, x_pre, ddof=1)[0, 1] / np.var(x_pre, ddof=1)
    return y - theta * (x_pre - x_pre.mean())

y_adj = cuped_adjust(df["margin_test"].to_numpy(), df["margin_pre"].to_numpy())
effect = y_adj[df["arm"] == "B"].mean() - y_adj[df["arm"] == "A"].mean()

Because x_pre was measured before assignment, it’s independent of treatment, so the estimate stays unbiased. Variance falls by the squared correlation between x_pre and y. For revenue metrics with stable customers, reductions of 30 to 60% are common, which can cut test duration roughly in half. Regression adjustment with several pre-period covariates generalizes the idea.

Stratified randomization helps too: randomize within strata (plan type, region, value band) so arms are balanced on the variables that matter most.

Delayed outcomes

Retention at day 90, default at 18 months, lifetime value over years. Options:

• Wait. Sometimes correct, especially for high-stakes policy changes. Run the test, then wait for outcomes to mature before reading it.
• Surrogate metrics. Use a short-term metric validated to predict the long-term outcome (14-day engagement for 90-day retention). Validate on historical experiments where both are known: did the surrogate’s effects predict the long-term effects?
• Ramp, then hold. Read early surrogates to decide whether to ramp up, but keep a long-term holdout running to confirm the real outcome later.

Peeking and sequential testing

Teams want to watch tests and stop them early when results look clear. With a standard fixed-horizon test, checking repeatedly and stopping at the first p < 0.05 inflates the false positive rate dramatically; daily peeking over a month can push it several times above the nominal 5%.

Either:

• Commit to a fixed horizon and don’t act on interim results, or
• Use a sequential method built for continuous monitoring: group sequential designs with pre-planned interim looks, or always-valid inference (mixture sequential probability ratio tests, confidence sequences). Most modern experimentation platforms support one of these.

Guardrail metrics are different: monitor them continuously and stop for harm. Stopping early because something is hurting customers is always allowed.

Ramps and rollout

A policy test is also a deployment:

1. Shadow mode: the candidate computes decisions alongside the current policy without acting. Compare action mixes, check guardrails, catch bugs. (Lesson 23 covers this.)
2. Small ramp (1 to 5%): real actions, watching guardrails.
3. Test allocation (often 50/50, or 10 to 20% for risky changes): read the primary metric.
4. Rollout with a holdout kept running to measure long-term value.

Log the policy version on every decision, so analysis can always tell which policy produced which outcome, even as ramps change.

Experiments tell you whether a policy is better today. The next lesson deals with what happens after: drift, feedback loops, and the slow decay of decision systems left alone.