Measuring Decision Quality, Not Model Accuracy

At some point, someone with budget authority asks: “Is the decision system working? What is it worth?” Most teams answer with model metrics: AUC is 0.87, calibration error is 0.01. That’s not an answer. The question was about decisions and money, and this lesson shows how to answer it properly.

Decision quality vs outcome luck

First, a distinction that keeps reviews sane. A decision is good if, given what was known at the time, it was the best available choice. An outcome is what then happened, which includes luck. A correctly blocked transaction with P(fraud) = 0.6 that turns out legitimate isn’t a bad decision; 40% of such blocks should turn out legitimate.

Judging decisions one at a time by their outcomes (“resulting”) leads to overreaction: a single high-profile false positive gets a threshold changed and costs far more over the next year than the incident did. Measure decision quality in aggregate, over enough decisions that luck averages out, and against alternatives.

The counterfactual problem, again

The obvious metrics are outcome rates for decisions made: retention among customers who got offers, fraud rate among approved transactions, default rate among approved loans. On their own, these say almost nothing about the decision system’s value, because they mix two things: what the policy did, and whom it chose to act on.

Retention of 91% among offer recipients is great if they’d have been at 70% without the offer, and terrible if they’d have been at 92%. The value of a decision policy is always relative to a counterfactual: what would have happened under a different policy.

Incremental value against a holdout

The most credible measurement uses the permanent randomized holdout from Lesson 10. A random share of units is decided by a baseline policy (often “do nothing”, or the old rule), and the rest by the current system. The difference in outcomes, scaled to the full population, is the system’s incremental value:

import numpy as np

def incremental_value(df, value_col="net_margin_12m", group_col="group"):
    """df: one row per unit. group in {'system', 'holdout'}, assigned
    at random. net_margin_12m includes offer and contact costs."""
    sys = df.loc[df[group_col] == "system", value_col].to_numpy()
    hold = df.loc[df[group_col] == "holdout", value_col].to_numpy()
    diff = sys.mean() - hold.mean()

    # Bootstrap CI: value per unit is noisy; never report a bare point estimate.
    rng = np.random.default_rng(0)
    boots = [rng.choice(sys, len(sys)).mean() - rng.choice(hold, len(hold)).mean()
             for _ in range(2000)]
    lo, hi = np.percentile(boots, [5, 95])
    n_units = len(df)
    return {"per_unit": diff, "ci90": (lo, hi), "total": diff * n_units}

Three requirements:

• The value column is the objective, including costs: net margin, not retention rate.
• Assignment is random and decided before anything else about the unit. Holdout units still pass through eligibility rules, so the comparison is between policies, not populations.
• Report intervals. Incremental value per unit is a difference of noisy averages. A point estimate without an interval invites overconfidence in both directions.

Where a permanent holdout is impossible, use the off-policy estimates of Lesson 15 against the baseline policy, or a periodic randomized test (next lesson). Label clearly which method produced the number.

Regret: how much is left on the table

Incremental value says how much better the system is than the baseline. Regret says how much better it could be. Compare the current policy’s value with:

• An oracle: the best decision in hindsight, where computable (inventory, staffing, routing, and other problems where you can replay outcomes). This is the ceiling.
• Strong alternatives: candidate policies evaluated offline (Lesson 15).
• Simple baselines: “do nothing”, “treat everyone”, the old rule. This is the floor.

Placing the current policy between floor and ceiling turns a vague question (“could the model be better?”) into a prioritization: if the policy captures 85% of the oracle’s value, the remaining 15% is the most any improvement could add, and you can weigh it against the engineering cost.

The decision scorecard

A decision system needs a small set of metrics that its owner reviews regularly. A good scorecard covers:

Model metrics belong on the scorecard, but in the “health” rows: they’re diagnostics that warn of problems before value drops, not the measure of value itself.

Look at segments

An overall improvement can hide segments where the policy loses money. Break down incremental value (and its interval) by the segments the business cares about: new vs established customers, plan type, region, channel, product line, risk band.

Patterns to look for:

• Segments with negative incremental value. Often sleeping dogs (Lesson 11) or segments with little training data where estimates are poor. Consider excluding them from treatment or exploring more there.
• Segments where the value is concentrated. If 80% of the value comes from 10% of decisions, that’s where model and data quality matter most.
• Segments that differ by group. The beginning of a fairness review.

Keep the number of segments modest and decide them in advance. Slicing a noisy metric twenty ways guarantees that some slices look alarming by chance.

Attribution when several systems act

Real customers are touched by many systems: a retention offer, a pricing change, a product email campaign. When retention rises, which one gets the credit? Holdouts per system answer this cleanly: each system’s holdout tells you its incremental value given everything else that was running. Without them, attribution becomes a negotiation between teams. It’s one more reason the holdout is the most valuable piece of infrastructure in this course.

Holdouts measure the running policy. To decide whether to switch to a new policy, you need a well-designed experiment, which is next.