SHAP (SHapley Additive exPlanations)

SHAP (SHapley Additive exPlanations) is a framework, introduced by Scott Lundberg and Su-In Lee at NeurIPS 2017, for answering one question about a single prediction: how much did each input feature contribute to this output, relative to what the model usually predicts? SHAP answers it by treating the features as players in a cooperative game whose payout is the prediction, and splitting that payout using Shapley values, a solution concept from game theory (Lloyd Shapley, 1953) that is the only way to divide the credit while satisfying a handful of fairness properties at once. The result is a set of signed numbers, one per feature, that sum exactly to prediction minus average prediction. SHAP has become the default explainability tool for tabular machine learning and is increasingly applied to text and model-reasoning attribution as well.

From Shapley Values to Feature Attributions

Imagine the features cooperating to produce a prediction. A coalition is any subset of features that are “present”; the ones outside the coalition are replaced by values drawn from a background dataset, so the model still returns something. Define v(S) as the model’s expected output when only the features in set S are known. The marginal contribution of feature i to a coalition S is v(S ∪ {i}) - v(S): how much the prediction moves when i joins.

Feature i’s Shapley value is that marginal contribution averaged over every possible coalition, weighted so that coalition sizes count equally:

φ_i = Σ  [ |S|! (n - |S| - 1)! / n! ] · ( v(S ∪ {i}) - v(S) )
     S ⊆ N\{i}

Equivalently, it is the average of i’s marginal contribution over all n! orderings in which the features could be added one at a time. SHAP frames this as an additive feature attribution:

f(x) ≈ φ_0 + Σ φ_i        where  φ_0 = E[f(X)]  (the base value)

# Scenario: the exact Shapley value for a small model, by brute force over
# all coalitions. This is the definition; it is exponential and only usable
# for a handful of features.
from itertools import combinations
from math import factorial

def shapley(v, n, i):
    others = [j for j in range(n) if j != i]
    total = 0.0
    for k in range(len(others) + 1):
        for S in combinations(others, k):
            w = factorial(k) * factorial(n - k - 1) / factorial(n)
            total += w * (v(frozenset(S) | {i}) - v(frozenset(S)))
    return total

The Guarantees SHAP Inherits

Shapley values are the unique attribution satisfying all of the following, which is the reason SHAP is preferred over ad hoc importance scores:

Consistency is the practically important one: it means you can compare SHAP values across model versions and trust the direction of the change.

Why You Need Approximations: the 2^n Problem

The definition sums over all 2^n coalitions (or all n! orderings). That is fine for five features and hopeless for fifty. Every practical SHAP method is an approximation or a model-class-specific exact shortcut. The most general approximation is permutation sampling: draw random feature orderings, compute the marginal contribution of each feature in each, and average. The estimate converges at the Monte Carlo rate, so the error shrinks like 1 / sqrt(m) in the number of samples m. The widget lets you watch that happen.

The SHAP Family

The shap library bundles one general estimator and several fast exact methods for specific model classes:

# Scenario: no tree structure to exploit, so fall back to the model-agnostic
# estimator with an explicit background sample.
background = shap.sample(X_tr, 100)                 # reference distribution
kernel = shap.KernelExplainer(model.predict_proba, background)
sv_row = kernel.shap_values(X_te.iloc[0, :], nsamples=500)   # 500 sampled coalitions

# Scenario: explaining a Hugging Face sentiment pipeline. PartitionExplainer
# groups adjacent tokens so correlated word-pieces are perturbed together.
import shap, transformers
clf = transformers.pipeline("sentiment-analysis")
explainer = shap.Explainer(clf)                     # picks PartitionSHAP for text
sv_text = explainer(["The plot dragged but the ending saved it."])
shap.plots.text(sv_text[0])                         # per-token push toward pos/neg

Local Explanations, Then Global Structure

A single row of SHAP values is a local explanation: base value, plus each feature’s signed push, equals this prediction. Stack the local explanations for a whole dataset and you get global structure without giving up local faithfulness:

• Mean absolute SHAP value per feature is a consistent global importance ranking (the shap.plots.bar view).
• The beeswarm plot shows, for every feature, the distribution of SHAP values across all rows colored by feature value, which reveals direction and non-linearity at a glance.
• Dependence plots put a feature’s value on the x-axis and its SHAP value on the y-axis, exposing thresholds and interactions.
• SHAP interaction values split a feature’s attribution into a main effect plus pairwise interaction terms, the local-interaction contribution from the 2020 trees paper.

# Scenario: turn many local explanations into a global picture.
import numpy as np
global_importance = np.abs(sv.values).mean(axis=0)   # per-feature mean |SHAP|
order = np.argsort(global_importance)[::-1]
shap.plots.beeswarm(sv)                               # direction + spread per feature

Pitfalls

• Correlated features. KernelSHAP’s default assumes features are independent, so it evaluates coalitions with unrealistic combinations (a 2-bedroom house with 6 bathrooms) and can spread credit onto features that only look important by correlation. PartitionSHAP and TreeSHAP’s tree_path_dependent option mitigate this; interventional TreeSHAP trades it for a different bias.
• Interventional vs conditional. “Interventional” perturbation breaks feature dependencies (true to the model, follows Shapley axioms exactly); “conditional” respects the data distribution (true to the data, but no longer strictly additive). They can disagree, and the library will not warn you which question you asked.
• Base value drift. The base value is the mean prediction over whatever background you pass. Change the background and every SHAP value shifts. Explanations are only comparable against a fixed reference.
• Not causal. A SHAP value says how the model uses a feature, not whether that feature causes the outcome. High SHAP importance for a proxy variable is a finding about the model, not the world.
• Adversarially foolable. Slack et al. (2020) showed a biased classifier can be wrapped in “scaffolding” that detects the out-of-distribution inputs SHAP and LIME generate and returns innocuous behavior on exactly those, hiding the bias from the explanation. Post hoc explanations are evidence, not proof.

What’s New (2024-2026)

• SHAP for LLM and transformer attribution. shap.Explainer on text now routes through PartitionSHAP by default, and teams use it to attribute a model’s output to spans of a prompt or a retrieved document, bringing SHAP into RAG and agent debugging rather than only tabular ML.
• Regulatory pull. The EU AI Act’s transparency obligations for high-risk systems, and similar model-risk-management expectations in finance, have made per-decision attributions a compliance artifact, and SHAP is the most common way teams produce them.
• Faster exact methods. GPU TreeSHAP and vectorized implementations in XGBoost and LightGBM made whole-dataset SHAP a routine step in model validation instead of an offline analysis.
• Known-limitations literature matured. Follow-up work formalized when interventional and conditional SHAP diverge, how correlated features distort attributions, and how to detect scaffolding-style attacks, so SHAP is now used with explicit caveats rather than treated as ground truth.

SHAP vs. LIME

LIME is the faster sketch; SHAP is the version with guarantees. In practice SHAP has largely displaced LIME for tabular models because TreeSHAP made the exact computation cheap, and because the additivity and consistency properties are what auditors and reviewers actually want.

# Scenario: a loan-scoring XGBoost model rejected an application and the
# reviewer needs to see which features pushed the score down.
import shap, xgboost
from sklearn.model_selection import train_test_split

X, y = shap.datasets.adult()               # census income data
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
model = xgboost.XGBClassifier(n_estimators=400, max_depth=4).fit(X_tr, y_tr)

# TreeExplainer computes exact SHAP values for tree ensembles in polynomial time
explainer = shap.TreeExplainer(model)
sv = explainer(X_te)                        # sv.values: (n_rows, n_features)

# Local explanation for a single applicant
shap.plots.waterfall(sv[0])                 # base value + each feature's push -> f(x)

# Global importance: mean absolute SHAP value per feature
shap.plots.bar(sv)