Expected Value & Cost-Sensitive Thresholds

Ask how a fraud threshold, a credit cutoff or a churn-risk trigger was chosen, and the honest answer is usually “someone picked 0.5” or “we tuned it until the review queue was manageable”. That’s a decision being made with no reference to what errors cost.

With calibrated probabilities from the last lesson, you can do better: derive the threshold from costs. This lesson builds that machinery, first for yes/no decisions, then for any number of actions.

The cost matrix

For a binary decision, there are four outcomes. Assign a value (or cost) to each:

The numbers come from the business, not the model. ”−$15” for a false block includes support contact cost, a share of the customers who abandon after being declined, and their lost future margin. Getting these estimates roughly right matters more than getting the model’s fourth decimal place right.

The optimal threshold

With a calibrated probability p of fraud, the expected cost of each action is:

• Block: (1 − p) × 15
• Approve: p × 400

Block when blocking is cheaper: p × 400 > (1 − p) × 15, which gives p > 15 / (15 + 400) ≈ 0.036.

In general, when correct decisions cost nothing, act when:

p > C_FP / (C_FP + C_FN)

where C_FP is the cost of acting wrongly (false positive) and C_FN is the cost of not acting when you should have (false negative). When correct decisions have non-zero values (a true positive earns something), use the general version: act when p × (value of TP − value of FN) > (1 − p) × (value of TN − value of FP).

Two things to notice. First, this threshold is only optimal if p is calibrated; on uncalibrated scores it’s meaningless, which is why Lesson 4 came first. Second, with asymmetric costs the threshold is far from 0.5. Teams that use a default predict() call (which thresholds at 0.5) are, for fraud, letting through most of the fraud the model correctly identifies as risky.

Per-instance costs

Costs often vary per decision. A fraudulent $2,000 purchase loses $2,000; a fraudulent $20 one loses $20. So the threshold should vary too:

import numpy as np

FRICTION = 15.0   # cost of blocking a legitimate customer

def should_block(p_fraud: np.ndarray, amount: np.ndarray) -> np.ndarray:
    # Expected loss if approved vs expected cost if blocked.
    # Equivalent to a per-row threshold of FRICTION / (FRICTION + amount).
    return p_fraud * amount > (1 - p_fraud) * FRICTION

# A $20 order needs p > 0.43 to block; a $2,000 order only p > 0.0074.

This is why “one threshold for everything” is almost never optimal. Any time the stakes of a decision vary across units (loan size, order value, customer lifetime value), the decision rule should use them directly.

From thresholds to expected-value argmax

Most real decisions have more than two actions. The retention example has five. The generalization is direct: for each eligible action, compute its expected value, and pick the largest.

from dataclasses import dataclass

@dataclass
class Offer:
    name: str
    cost_if_accepted: float   # e.g. discount value over 3 months
    contact_cost: float

OFFERS = [
    Offer("none", 0.0, 0.0),
    Offer("discount_10pct_3mo", 0.10 * 3 * 40, 0.05),
    Offer("discount_25pct_3mo", 0.25 * 3 * 40, 0.05),
]

def expected_value(p_retain: float, offer: Offer, clv: float) -> float:
    # Value of a retained customer minus what the offer costs.
    # Simplification: the discount is paid only if they stay.
    return p_retain * (clv - offer.cost_if_accepted) - offer.contact_cost

def best_offer(customer, model, clv: float) -> tuple[str, float]:
    # model.p_retain(customer, offer): calibrated P(retained at day 90
    # | customer, offer). Estimating this per offer is a causal question,
    # which Module 4 tackles properly.
    scored = [(o.name, expected_value(model.p_retain(customer, o), o, clv))
              for o in OFFERS]
    return max(scored, key=lambda t: t[1])

Including “none” is crucial. The value of an offer is not p_retain × CLV; it’s the difference between its expected value and doing nothing. A customer with p_retain = 0.95 under no offer and 0.96 with a 25% discount should get nothing: the discount buys one percentage point of retention at the cost of 25% off for everyone who would have stayed anyway.

This is also where the prediction requirement becomes clear. You need P(outcome | action) for each action, not just one churn probability. Estimating that properly is the subject of Module 4. For now, notice that a single churn score can’t fill in this function.

Sensitivity analysis: showing stakeholders their numbers

Cost estimates are uncertain, and stakeholders disagree. Instead of arguing over the exact false-positive cost, compute what each plausible value implies:

def profit_at(threshold, p, y, c_fp, c_fn):
    act = p >= threshold
    return -(np.sum(act & (y == 0)) * c_fp + np.sum(~act & (y == 1)) * c_fn)

for c_fp in [5, 10, 15, 30]:
    t = c_fp / (c_fp + 400)
    blocked = np.mean(p_val >= t)
    print(f"C_FP=${c_fp}: threshold={t:.3f}, block rate={blocked:.1%}, "
          f"profit={profit_at(t, p_val, y_val, c_fp, 400):,.0f}")

A table like this changes the conversation. Often the profit curve is flat across the plausible range and the argument evaporates. When it isn’t, the owner can see exactly what they’re choosing: “at $30 we block 0.8% of customers; at $10 we block 2.1% and catch 40% more fraud.”

Plot expected profit against threshold on held-out data as a standard artifact for every decision model. The curve shows the optimum, how flat it is around the optimum, and how much you lose by being a bit off. It’s far more useful in a review than an ROC curve.

Capacity: when the threshold is set by operations

Sometimes the real constraint is capacity: the fraud team can review 2,000 cases a day, or the call centre can make 600 retention calls a month. Then the decision isn’t “is this case above the cost threshold?” but “which 2,000 cases are most worth reviewing?”.

For a single capacity constraint, rank by expected value of acting (not by probability) and take the top k. That’s already a tiny optimization problem. With several constraints and actions competing for them, you need a real solver, which is Module 3.

Before that, one more gap in the predictions: point estimates hide how sure the model is. The next lesson adds uncertainty intervals with guarantees.