Causal Effects from Observational Data

The last lesson assumed randomized data. Often you don’t have it: the programme ran for years under a targeting policy, randomizing now would take a quarter to read out, or randomizing is unethical or illegal for the decision at hand. You still need effect estimates.

You can get them from observational data, under assumptions. This lesson covers the methods, the assumptions they rest on, and the checks that keep you honest. Treat everything here as weaker evidence than a randomized test, and validate against randomized data whenever you can get some.

The assumptions

Estimating causal effects from logs requires three things:

1. No unmeasured confounding (also called ignorability or exchangeability): every variable that influenced both the treatment decision and the outcome is in your data. This is untestable from data alone. It’s a claim about how the historical decisions were made, which is why the DAG conversation in Lesson 10 matters.
2. Overlap (positivity): for every type of unit, there was some chance of both receiving and not receiving the treatment. If the old policy always called customers with engagement above 2, the data contains no information about what happens when you don’t call them.
3. Consistency: the treatment is well defined. “Got a retention offer” covering five different offers isn’t one treatment.

Everything below is machinery for using these assumptions efficiently. None of it rescues you if they fail.

Propensity scores and weighting

The propensity score e(x) = P(T = 1 | X = x) is the probability of being treated given the features. If the confounders are all in X, then within groups of equal propensity, treatment is as good as random.

Inverse propensity weighting (IPW) reweights the data so the treated and control groups look like the whole population: each treated unit counts 1/e(x), each control 1/(1 − e(x)).

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier

# X: confounders (pre-treatment features only!), t: 0/1 treatment, y: outcome
ps_model = HistGradientBoostingClassifier(max_iter=300).fit(X, t)
e = ps_model.predict_proba(X)[:, 1]

# Overlap check: look at the propensity distributions per group.
print("treated e range:", np.percentile(e[t == 1], [1, 50, 99]))
print("control e range:", np.percentile(e[t == 0], [1, 50, 99]))

# Trim to the region of common support, then weight.
keep = (e > 0.02) & (e < 0.98)
w = np.where(t == 1, 1 / e, 1 / (1 - e))[keep]
ate_ipw = (np.sum(w * y[keep] * t[keep]) / np.sum(w * t[keep])
           - np.sum(w * y[keep] * (1 - t[keep])) / np.sum(w * (1 - t[keep])))

Two rules that save teams from embarrassing results:

• Only pre-treatment features. Anything measured after the treatment (including things the treatment influenced) must not be in X. This is the most common leakage bug in causal pipelines.
• Check overlap before estimating. If propensities pile up near 0 or 1, some units carry enormous weights and the estimate becomes a coin toss. Trim to common support and be explicit that the estimate only applies to that population.

The trimming threshold changes the population you’re describing. That’s fine, as long as you say so: “among customers with between 2% and 98% chance of being called under the old policy, calling changes retention by…”.

Doubly robust estimation

IPW depends entirely on the propensity model; outcome regression depends entirely on the outcome model. Doubly robust estimators (augmented IPW) combine them: fit outcome models μ₁(x), μ₀(x) and a propensity model e(x), and estimate

τ̂ = mean[ μ₁(x) − μ₀(x)
          + t·(y − μ₁(x))/e(x)
          − (1 − t)·(y − μ₀(x))/(1 − e(x)) ]

If the outcome model is right, the correction terms average to zero. If the propensity model is right, they correct the outcome model’s bias. The estimate is consistent if either is correct, and in practice it’s usually more stable than either alone. You’ll meet the same estimator again for off-policy evaluation in Lesson 15.

Fit the nuisance models with cross-fitting: train on one fold, predict on another, so each unit’s predictions come from models that didn’t see it. Without it, flexible ML models overfit and bias the estimate.

Double machine learning and causal forests

For targeting you need heterogeneous effects (CATE), with uncertainty. Double machine learning (DML) (Chernozhukov et al., 2018) does this by “partialling out”: predict the outcome from confounders, predict the treatment from confounders, and regress the outcome residuals on the treatment residuals. What’s left is the effect, free of the confounders’ influence, with valid confidence intervals even when the nuisance models are flexible ML.

Causal forests combine this with a random-forest-style estimator of how the effect varies with features. EconML implements both:

from econml.dml import CausalForestDML
from sklearn.ensemble import HistGradientBoostingRegressor, HistGradientBoostingClassifier

est = CausalForestDML(
    model_y=HistGradientBoostingRegressor(),   # outcome nuisance model
    model_t=HistGradientBoostingClassifier(),  # propensity nuisance model
    discrete_treatment=True,
    cv=5,                                      # cross-fitting folds
    random_state=0,
)
# X: features the effect may vary with; W: extra confounders to adjust for
est.fit(Y=y, T=t, X=X_effect, W=W_confounders)

cate = est.effect(X_new)                         # per-customer uplift
lo, hi = est.effect_interval(X_new, alpha=0.1)   # 90% intervals

The intervals are useful in decisions. Customers whose effect interval excludes zero can be targeted confidently; those whose interval straddles zero are candidates for exploration (Module 5) rather than confident action.

Refute before you trust

Observational estimates can be wrong in ways no metric reveals. DoWhy structures the workflow as model (write the DAG), identify (derive what to adjust for), estimate, and refute:

from dowhy import CausalModel

model = CausalModel(data=df, treatment="called", outcome="retained",
                    graph=dag_string)          # the DAG from Lesson 10
estimand = model.identify_effect()
estimate = model.estimate_effect(estimand,
                                 method_name="backdoor.propensity_score_weighting")

# Placebo: a random treatment should have ~zero effect.
model.refute_estimate(estimand, estimate, method_name="placebo_treatment_refuter")
# Adding a random common cause should not move the estimate much.
model.refute_estimate(estimand, estimate, method_name="random_common_cause")
# Re-estimating on subsets should give similar results.
model.refute_estimate(estimand, estimate, method_name="data_subset_refuter")

Also run a sensitivity analysis: how strong would an unmeasured confounder have to be to explain away the effect? If a confounder as strong as tenure would erase it, be cautious. If it would need to be ten times stronger than anything you’ve measured, the estimate is sturdier.

And the strongest check of all: compare against a randomized benchmark. If your permanent holdout (Lesson 10) covers even a slice of the population, compare observational and randomized estimates on that slice. Large disagreement means the observational pipeline is missing something.

Natural experiments in your own rules

Decision systems are full of accidental experiments. Look for them:

• Cutoffs → regression discontinuity. Approved at credit score ≥ 680, free shipping above $50, VIP status at 10 orders. Units just either side of the cutoff are nearly identical, so the jump in outcomes at the cutoff estimates the effect locally.
• Rollouts → difference-in-differences. A new policy launched in some regions before others. Compare the change over time in launched regions with the change in not-yet-launched ones.
• Arbitrary assignment. Cases routed to reviewers by queue order, offers capped when the daily budget ran out at a random time of day. If assignment was effectively random for reasons unrelated to outcomes, it’s an experiment.

These designs often give more credible estimates than any amount of propensity modeling, because their assumptions are easier to defend. They also suggest a design principle: when you write a new hard threshold into a decision system, you’re creating a future natural experiment. Log the running variable.

With effects estimated, by randomization or carefully from logs, the decision stack can finally target the right customers. But there’s a gap: what about actions you’ve never tried, and worlds that keep changing? That’s Module 5.