Uncertainty with Conformal Prediction & Quantiles

A demand forecast of 820 units says nothing about whether demand could plausibly be 400 or 1,400. A loss estimate of $3,000 doesn’t tell you whether you’re nearly sure or barely guessing. Decisions need that information. Order quantities, safety buffers, credit limits and “should a human check this?” all depend on how uncertain the prediction is, not just its central value.

This lesson covers two practical tools: quantile models, which predict a chosen point of the outcome distribution directly, and conformal prediction, which wraps any model with intervals or sets that come with a statistical guarantee.

When the decision needs a quantile, not a mean

The classic example is the newsvendor problem. You must choose a quantity before demand is known. Each unit short costs you the lost margin (underage cost Cu). Each unit left over costs you the waste (overage cost Co). The optimal quantity is the demand quantile at the critical ratio:

q* = F⁻¹( Cu / (Cu + Co) )

If a pastry sells for $4, costs $1, and leftovers are binned, Cu = $3 and Co = $1, so produce the 75th percentile of demand. For a high-margin item with cheap storage, the ratio might be 0.95; for perishable low-margin stock, 0.3. Ordering the mean forecast is only right when the two costs are equal, which they almost never are.

The same structure appears in staffing (understaffing vs idle cost), cash buffers, cloud capacity and safety stock. Whenever errors in one direction cost more than in the other, the decision wants a quantile.

You can predict a quantile directly by training with the pinball (quantile) loss:

from sklearn.ensemble import HistGradientBoostingRegressor

Cu, Co = 3.0, 1.0
tau = Cu / (Cu + Co)    # 0.75

# Pinball loss at tau penalizes under-prediction tau/(1-tau) times
# more than over-prediction, so the model learns the tau-quantile.
q_model = HistGradientBoostingRegressor(loss="quantile", quantile=tau, max_iter=300)
q_model.fit(X_train, demand_train)

order_qty = np.ceil(q_model.predict(X_next_day))

# Sanity check on held-out data: about 75% of days should have
# demand at or below the predicted quantity.
print((demand_val <= q_model.predict(X_val)).mean())

That last check matters. Quantile models are not guaranteed to hit their nominal coverage, especially at extreme quantiles or with little data. Conformal prediction is how you get a guarantee.

Split conformal prediction

Conformal prediction takes any trained model and a held-out calibration set, and produces intervals (for regression) or sets (for classification) that contain the truth with a chosen probability, say 90%. It makes no assumptions about the model being correct. It only assumes the calibration data and future data are exchangeable, roughly “drawn the same way”.

The recipe for regression:

1. Train the model on the training set.
2. On the calibration set, compute a nonconformity score for each point: here, the absolute error |y − ŷ|.
3. Take the score quantile at level ⌈(n+1)(1−α)⌉/n.
4. For a new point, the interval is ŷ ± that quantile.

import numpy as np

def conformal_quantile(scores: np.ndarray, alpha: float) -> float:
    n = len(scores)
    # Finite-sample correction: the test point is one more
    # exchangeable draw, so use the ceil((n+1)(1-alpha))-th score.
    level = min(1.0, np.ceil((n + 1) * (1 - alpha)) / n)
    return np.quantile(scores, level, method="higher")

# 1. model already trained on X_train
# 2. scores on a calibration split the model never saw
cal_scores = np.abs(y_cal - model.predict(X_cal))
qhat = conformal_quantile(cal_scores, alpha=0.1)

# 3. 90% prediction intervals for new points
pred = model.predict(X_new)
lower, upper = pred - qhat, pred + qhat

That’s all there is to it: a few lines, any model, and a guarantee that at least 90% of future intervals cover the truth (on average, under exchangeability).

The plain version gives every point the same interval width. Conformalized quantile regression (CQR) fixes that: train lower and upper quantile models, compute scores as how far each calibration point falls outside its predicted band, and widen both quantile predictions by the conformal quantile. You get intervals that are wide where the data is noisy and narrow where it isn’t, with the same guarantee.

Conformal prediction sets for classification

For classifiers, conformal prediction outputs a set of labels instead of one. A simple score is 1 minus the probability the model assigns to the true class:

# probs_cal: (n, K) predicted class probabilities, y_cal: true labels
cal_scores = 1 - probs_cal[np.arange(len(y_cal)), y_cal]
qhat = conformal_quantile(cal_scores, alpha=0.05)

# Include every class whose score is within the threshold.
pred_sets = probs_new >= (1 - qhat)    # boolean (m, K) matrix
set_sizes = pred_sets.sum(axis=1)

Most inputs get a single label. Ambiguous ones get two or three. Inputs the model has never seen anything like may get all of them. Set size is an honest uncertainty signal, and it maps cleanly onto decisions:

• Set size 1: automate the decision.
• Set size 2 or more: route to a human or a more expensive check.
• Empty set (possible with some scores): treat as out-of-distribution and escalate.

Changing α trades automation rate against error rate, and the trade-off is predictable in advance, which is exactly what a decision owner wants.

What the guarantee does and doesn’t cover

Conformal guarantees are real but narrow. Be precise about them in design reviews:

• Marginal, not conditional. 90% coverage is averaged over all future points. Some subgroups (new customers, a small region) can be well below 90% while others are above. Mondrian conformal computes a separate quantile per group and gives coverage per group, at the cost of needing enough calibration data in each.
• Exchangeability, not shift. If the future differs from the calibration data (a new product line, a pandemic, a fraud attack), coverage can break. Recalibrate on a rolling window of recent data, or use adaptive conformal methods that adjust α online based on observed misses.
• Coverage, not usefulness. An interval of [0, ∞) has 100% coverage. Report average width or average set size next to coverage; a model with better underlying predictions gives tighter sets for the same guarantee.

Using uncertainty in decisions

Uncertainty estimates earn their keep when they change actions:

• Size the buffer. Order to the quantile the costs demand; hold safety stock proportional to interval width.
• Abstain or escalate. Automate when the interval or set is tight, send to review when it’s wide. This is often the single biggest practical win: automate the easy 80% confidently, put human attention where the model is unsure.
• Be conservative where errors are expensive. Approve a credit limit based on the lower bound of predicted income, not the mean.
• Feed robust optimization. Intervals become the uncertainty sets that Lesson 9’s robust optimization protects against.

With calibrated probabilities, cost-derived thresholds and honest uncertainty, the prediction side of the stack is ready. Next module: what to do when decisions compete for limited resources.