Probability Calibration

A decision rule multiplies a probability by an amount of money. “Block this transaction if P(fraud) × $400 > $15.” That rule is only as good as the probability going into it. If the model says 0.05 for transactions that are fraudulent 10% of the time, every decision built on it is systematically wrong, and no amount of AUC will tell you.

Calibration is the property that predicted probabilities match observed frequencies. Among all the cases where the model says 30%, about 30% should turn out positive. It’s the single most important property a model needs to be useful inside a decision, and one of the least checked.

Discrimination vs calibration

There are two separate questions you can ask of a probabilistic classifier:

• Discrimination: does it rank positives above negatives? Measured by AUC, precision at k, lift.
• Calibration: are its probabilities numerically right? Measured by reliability diagrams, expected calibration error, and (partly) the Brier score and log loss.

These are independent. A model can rank perfectly and be badly calibrated: apply any monotonic squashing to its outputs and AUC stays the same while the probabilities become nonsense. A model can also be well calibrated and rank poorly: predicting the base rate for everyone is perfectly calibrated and useless.

Ranking is enough if your decision is “take the top 1,000”. The moment the decision compares a probability against costs, calibration matters.

Where miscalibration comes from

Most production models are miscalibrated for predictable reasons:

• Class rebalancing. Oversampling, undersampling, SMOTE or class weights change the prior the model sees, so its probabilities are inflated for the minority class. This is the most common cause by far.
• Model family. Random forests tend to push probabilities away from 0 and 1. Naive Bayes pushes them towards the extremes. Modern deep networks are often overconfident.
• Regularization and early stopping. These trade calibration against other goals, sometimes in surprising directions.
• Distribution shift. A model calibrated last year is miscalibrated today if the base rate has moved, even if its ranking is still fine.

Gradient-boosted trees and logistic regression trained on log loss with the natural class balance are usually close to calibrated. “Usually close” isn’t a guarantee, so measure.

Measuring calibration

The reliability diagram bins predictions and plots the mean predicted probability in each bin against the observed positive rate. A perfectly calibrated model sits on the diagonal.

import numpy as np
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss, log_loss

# p_val: model probabilities on a held-out set, y_val: true labels
frac_pos, mean_pred = calibration_curve(y_val, p_val, n_bins=10, strategy="quantile")

def expected_calibration_error(y, p, n_bins=10):
    # Weighted average gap between confidence and accuracy per bin.
    # Quantile bins keep every bin populated when scores are skewed.
    edges = np.quantile(p, np.linspace(0, 1, n_bins + 1))
    bins = np.clip(np.digitize(p, edges[1:-1]), 0, n_bins - 1)
    ece = 0.0
    for b in range(n_bins):
        mask = bins == b
        if mask.any():
            ece += mask.mean() * abs(p[mask].mean() - y[mask].mean())
    return ece

print("ECE:", expected_calibration_error(y_val, p_val))
print("Brier:", brier_score_loss(y_val, p_val))
print("Log loss:", log_loss(y_val, p_val))

Use quantile bins when probabilities are concentrated near zero, as they are for most rare-event problems; equal-width bins would put 95% of the data in the first bin and tell you nothing. And look at the region where you make decisions. If your fraud threshold sits around p = 0.04, calibration between 0.01 and 0.10 matters far more than calibration near 0.9.

The Brier score (mean squared error of probabilities) and log loss are proper scoring rules: they’re minimized by the true probabilities, so they reward both discrimination and calibration. They’re good single numbers for comparing models when you care about probability quality.

Fixing calibration

Recalibration learns a mapping from raw scores to calibrated probabilities, fit on data the base model didn’t see.

• Platt scaling fits a logistic regression on the score: two parameters, stable with small data, assumes a sigmoid-shaped distortion.
• Isotonic regression fits any monotonic step function: flexible, needs more data (think thousands of points with a healthy number of positives), can overfit small sets.
• Temperature scaling divides a neural network’s logits by one learned constant: the standard for deep nets, preserves ranking exactly.

scikit-learn’s CalibratedClassifierCV handles the data split for you with cross-validation:

from sklearn.calibration import CalibratedClassifierCV
from sklearn.ensemble import HistGradientBoostingClassifier

base = HistGradientBoostingClassifier(max_iter=400, learning_rate=0.05)

# cv=5: trains 5 base models on 4/5 of the data and fits a
# calibrator on the held-out 1/5 each time. Predictions average
# the 5 calibrated models. Never calibrate on training scores.
calibrated = CalibratedClassifierCV(base, method="isotonic", cv=5)
calibrated.fit(X_train, y_train)

p_val = calibrated.predict_proba(X_val)[:, 1]

If you already have a trained model you can’t refit (for example, it’s served by another team), fit a calibrator on its scores over a fresh held-out set yourself. It’s just a one-feature logistic regression or an IsotonicRegression, versioned alongside the model it corrects.

Correcting for rebalancing and base-rate shift

When you know the training prior differs from the deployment prior, you can correct analytically. If the model was trained where the positive rate was π_train and the real rate is π_true, Bayes’ rule gives:

def prior_shift(p, pi_train, pi_true):
    # Reweight the odds by the ratio of true to training priors.
    a = p * (pi_true / pi_train)
    b = (1 - p) * ((1 - pi_true) / (1 - pi_train))
    return a / (a + b)

# Trained on 50/50 rebalanced data, real churn rate is 5%:
prior_shift(0.6, pi_train=0.5, pi_true=0.05)   # -> about 0.073

That 0.6 “high risk” customer is really a 7% risk. Teams that skip this step build entire retention programmes on probabilities inflated tenfold.

The same formula handles base-rate drift in deployment: if fraud prevalence doubles during a holiday attack, the ranking may be fine but the probabilities are too low. Monitoring the observed positive rate and re-applying the correction (or refitting the calibrator on recent labels) keeps decisions honest between full retrains.

Calibration in production

Treat calibration as part of the model artifact:

• Version the calibrator with the model. Shipping a retrained model with last month’s calibrator is a classic source of silent decision shifts.
• Check calibration in the decision region on every retrain, as a release gate alongside AUC.
• Calibrate per segment when decisions differ by segment. A model calibrated overall can be miscalibrated for new customers or a particular country. If you make different decisions for those groups, check them separately. This also matters for fairness (Lesson 22).
• Monitor calibration drift once labels arrive, comparing predicted and realized rates per score bucket.

With probabilities you can trust, you can finally turn them into decisions that weigh costs properly. That’s the next lesson.