Predictions Are Not Decisions

Most machine learning projects end at the wrong place. The notebook finishes with a score: a churn probability, a fraud risk, a demand forecast. Someone wires that score to a threshold picked in a meeting, and the project is declared shipped. Six months later the business asks why the churn programme costs more than it saves, and nobody can answer, because nobody ever modeled the part that actually touches money: the decision.

This course is about that missing part. A prediction is a statement about the world. A decision is a commitment to act on it. The gap between them is where most of the value of applied ML is either captured or quietly lost.

Four ingredients of every decision

Strip any business decision down and you find the same four pieces. A loan approval, a replenishment order, a fraud block, a retention offer: all of them have this shape.

• Actions. The set of things you could do. Approve, decline, or approve at a lower limit. Order 0, 100, 500 or 900 units. Send no offer, a 10% offer, or a call from an account manager. If there is only one possible action, there is no decision.
• Outcomes. What happens in the world after you act. The loan is repaid or defaults. The SKU sells through or sits in the warehouse. The customer stays or leaves.
• Objective. How you score outcomes. Usually money (profit, cost, revenue at risk), sometimes a blend with risk, fairness or customer experience. It has to be explicit, because without it “better” means nothing.
• Uncertainty. You don’t know which outcome each action will produce. Prediction lives here, and only here. The model’s job is to describe how likely each outcome is given each action, so the decision layer can weigh them.

flowchart LR
    X["Context\ncustomer, item, time"] --> P["Predictive model\nP(outcome given action, context)"]
    A["Candidate actions\noffer A, offer B, nothing"] --> P
    P --> D["Decision layer\nmaximize expected objective\nsubject to constraints"]
    O["Objective and constraints\nmargin, budget, policy"] --> D
    D --> ACT["Chosen action"]
    ACT --> W["World\nobserved outcome"]
    W -.->|"logged for learning"| P

    style X fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style A fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style P fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style O fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style D fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style ACT fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style W fill:#EEF0F7,stroke:#6366F1,color:#0F172A

Look at where the predictive model sits in that picture: in the middle, as one input. It’s an important input. But the decision layer, the objective and the constraints are what turn it into an action, and those are usually built least carefully.

The churn programme that lost money

Here is the failure pattern in its most common form. A subscription business trains a churn classifier. It’s a good model, 0.91 AUC on a held-out month. The retention team sends a 20% discount to every customer with churn probability above 0.6.

Revenue drops. Three things went wrong, and none of them is the model’s accuracy.

First, risk is not responsiveness. The customers most likely to churn include many who will leave no matter what: they’ve moved country, their company was acquired, they’ve already signed with a competitor. The discount is wasted on them. Meanwhile, customers at 0.4 risk who would have been swayed by a small gesture get nothing. The model answered “who will churn?” when the decision needed “whose behaviour does this offer change?”. Those are different quantities, and Module 4 shows how to estimate the second one.

Second, the cost structure was never written down. A 20% discount given to someone who would have stayed anyway is a pure loss of 20% of their revenue, for the rest of their tenure. A discount that saves a customer earns their remaining lifetime value minus the discount. Without those numbers in the decision rule, the 0.6 threshold is a guess dressed up as data science.

Third, there was no constraint. The retention budget, the capacity of the account management team, and the rule that nobody gets two discounts in a year all exist, but none of them were in the system. The policy was enforced, when it was enforced at all, by people overriding the list in a spreadsheet.

The fix isn’t a better classifier. It’s a decision model: estimate the effect of each offer on each customer, value those effects in money, and choose offers to maximize expected profit within the budget and policy rules. That’s the system you’ll build in the capstone.

Why accuracy can mislead you

It feels obvious that a more accurate model should give better decisions. Often it does. But “accurate” is measured with metrics that ignore what the decision cares about.

# Two fraud models scored on the same 10,000 transactions.
# Model B has better AUC. Model A makes more money at the
# operating point the business actually uses.

import numpy as np
from sklearn.metrics import roc_auc_score

FRAUD_LOSS = 400.0     # average loss if fraud is approved
FRICTION_COST = 15.0   # cost of blocking a good customer (support, churn risk)

def expected_profit(y_true, scores, threshold):
    block = scores >= threshold
    missed_fraud = np.sum((~block) & (y_true == 1))
    blocked_good = np.sum(block & (y_true == 0))
    return -(missed_fraud * FRAUD_LOSS + blocked_good * FRICTION_COST)

# Evaluate both models at the threshold ops will run (say 0.5),
# not across all thresholds the way AUC does.
for name, s in [("A", scores_a), ("B", scores_b)]:
    print(name, roc_auc_score(y, s), expected_profit(y, s, 0.5))

AUC asks “how well does this model rank positives above negatives, averaged over every possible threshold?”. The business acts at one threshold, with costs that are wildly asymmetric: missing a fraud costs 400, blocking a good customer costs 15. Model B can rank better in regions you’ll never operate in and worse in the one region you do. AUC is also blind to calibration: multiply every score by 0.5 and AUC doesn’t move, but every threshold-based decision changes.

None of this means “ignore model metrics”. It means model metrics are diagnostics, and the decision metric, expected profit, cost or regret at the policy you’ll actually run, is the score that matters. Lesson 19 builds that measurement properly.

The anti-pattern: score and walk away

The most common architecture in industry looks like this: the data science team publishes a score to a feature table, and some downstream system (a CRM rule, a SQL job, a human with a spreadsheet) turns it into an action. Ownership of the decision is split across teams that never talk.

The symptoms are recognisable:

• Thresholds are constants in application code, last changed two years ago by someone who has left.
• Nobody can say what a 1-point lift in model AUC is worth in money.
• Business rules that override the model live in three different places and contradict each other.
• When the model is retrained, decisions shift unpredictably, because the score distribution moved and the thresholds didn’t.
• There is no log connecting “we took this action, for this reason” to “this happened afterwards”, so nothing can be learned from the decisions themselves.

Decision intelligence, the umbrella term this course uses, is the discipline of treating the decision as the product. The score is a component. The decision has an owner, an objective, a version, a log and an evaluation, just like any other production system.

What this course builds

Each module adds one layer of the decision stack, using a running example of a subscription business deciding which retention offer, if any, to give each customer:

1. Framing (this module): actions, objectives, constraints, and choosing the right technique.
2. Trustworthy predictions: calibrated probabilities, cost-sensitive thresholds, and uncertainty intervals with guarantees.
3. Optimization: choosing the best combination of actions under budgets and capacity.
4. Causality: estimating what an action changes, not just what will happen.
5. Sequential learning: bandits that learn while deciding, and evaluating new policies from old logs.
6. Rules and LLMs: decision tables, rules engines, and language models as components with guardrails.
7. Evaluation and monitoring: decision-quality metrics, experiments, and feedback loops.
8. Governance and engineering: fairness, explanations, audit trails, decision services and simulation.
9. Capstone: all of it assembled into one production retention offer engine.

The next lesson starts where every decision project should: writing down the decision before writing any model code.