Visual explainer
SHAP & LIME
Follow a weighted local fit and a two-feature Shapley game, then see how references and missing-feature assumptions change the explanation.
LIME describes a prediction with a nearby interpretable fit. SHAP allocates a prediction's difference from a baseline using a specified feature-coalition game. Both explain model behavior; neither turns feature credit into a causal promise.
Use the synthetic score on normalized features between zero and one. The query scores . The two rails are feature values, not feature importances. This is a worked model, not a loan decision or a measured benchmark.
Fit A Local Slice
Hold and perturb to . Model scores are . For an illustrative proximity kernel , their weights are . Dot area follows weight. Only the query itself has zero distance and weight one.
With an unpenalized intercept and slope, weighted least squares fits exactly because this particular slice is linear. This demonstrates one feature while fixing the other; a full two-feature explanation needs perturbations in both. General LIME fits are approximate, depend on the neighborhood and interpretable representation, and may include a complexity penalty. See the LIME paper, sections 3.2–3.4.
Account For The Difference
For the Shapley calculation, replace each absent feature with its value in the fixed reference . This defines a single-reference replacement game: , , , . A background distribution would instead average replacement evaluations. Conditional expectations define another possible missing-feature game; these choices need not agree.
The illustrated allocation is baseline , A credit , B credit . Horizontal lengths share a score scale, so the endpoint is exactly . Exact Shapley values satisfy this additive accounting for the specified game; numerical approximations can have residual error. The SHAP paper, sections 2–4 develops additive attributions and their defining properties.
Average Over Arrival Orders
There are two equally weighted arrival orders. If A arrives before B, its gain is . If B is present first, A adds . Thus . The two arrow lengths use the same score scale; the second is twice the first. This averaging splits the interaction term between the two features.
Rank This Prediction's Credits
B adds when first and when second. Its mean gain is . The bars rank A above B by credit magnitude for this query and reference. They do not rank the features globally, establish fairness, or prescribe an intervention. The allocation accounts for above baseline; it does not explain the baseline itself.
A Local Slope Can Change
The solid slice has slope . The dashed slice is , with slope . Both use identical axes. The interaction makes sensitivity depend on the held value of B. Neither slope equals A's Shapley credit here: a change per unit and an allocated score difference answer different questions. There is no general requirement that LIME coefficients equal SHAP credits.
Check What Masks Mean
Now add a support assumption: suppose observed feature pairs are only and , shown as solid dots. The dashed diagonal is a guide, not extra observations. Independently replacing just one feature queries the crossed or combinations. Those evaluations are defined for our formula but unsupported by the assumed data.
A conditional game or grouped features changes the question being asked. Neither choice automatically reveals causality. Report the masking convention and check whether the queried combinations represent plausible inputs.
Change Only The Reference
Return to the original formula and query , now using the single reference . The four coalition scores become . A receives . B receives .
The lower bar still ends at : . Same prediction, different comparison. Both rows use 560 drawing units per score unit. A reference is part of the explanation, not an invisible default.
Predict A Fresh Query
For with reference , first calculate the output: . The coalition scores are . A receives ; B receives . The bar checks . Reusing the old credits would explain the wrong prediction.
Where It Breaks
For a separate boundary example, compare two candidate models and . If observations contain only and , both models produce and . A filled dot and open square coincide on each observed row.
At the unsupported probe they separate: , . Observed agreement cannot determine which extension is appropriate. Attributions to a chosen model can faithfully describe that model while still failing to justify a real-world intervention. Evaluate the model, the reference and perturbation assumptions alongside the explanation.
The Quick Version
- A score alone does not allocate feature credit.
- LIME fits an interpretable model under a chosen neighborhood.
- SHAP credits depend on the missing-feature game and reference.
- Arrival-order averaging distributes interactions across features.
- Local slopes and allocated score differences are different quantities.
- Correlated-feature masks can leave observed support.
- New queries need new credits.
- Additive accounting does not establish causal recourse or fairness.