Skip to content
AI360XpertBeta
Core ML

Visual explainer

SHAP & LIME

Follow a weighted local fit and a two-feature Shapley game, then see how references and missing-feature assumptions change the explanation.

A score alone does not allocate credit to the features.
A score alone does not allocate credit to the features.

LIME describes a prediction with a nearby interpretable fit. SHAP allocates a prediction's difference from a baseline using a specified feature-coalition game. Both explain model behavior; neither turns feature credit into a causal promise.

Use the synthetic score f(a,b)=0.2+0.2a+0.1b+0.2abf(a,b)=0.2+0.2a+0.1b+0.2ab on normalized features between zero and one. The query (1,1)(1,1) scores 0.70.7. The two rails are feature values, not feature importances. This is a worked model, not a loan decision or a measured benchmark.

Fit A Local Slice

LIME can fit a proximity-weighted slice around one prediction.
LIME can fit a proximity-weighted slice around one prediction.

Hold b=1b=1 and perturb aa to 0.5,0.75,10.5,0.75,1. Model scores are 0.5,0.6,0.70.5,0.6,0.7. For an illustrative proximity kernel w=exp⁡(−d2/0.52)w=\exp(-d^2/0.5^2), their weights are 0.368,0.779,10.368,0.779,1. Dot area follows weight. Only the query itself has zero distance and weight one.

With an unpenalized intercept and slope, weighted least squares fits g(a)=0.3+0.4ag(a)=0.3+0.4a exactly because this particular slice is linear. This demonstrates one feature while fixing the other; a full two-feature explanation needs perturbations in both. General LIME fits are approximate, depend on the neighborhood and interpretable representation, and may include a complexity penalty. See the LIME paper, sections 3.2–3.4.

Account For The Difference

The baseline plus the two Shapley credits reconstructs the score.
The baseline plus the two Shapley credits reconstructs the score.

For the Shapley calculation, replace each absent feature with its value in the fixed reference (0,0)(0,0). This defines a single-reference replacement game: v(∅)=0.2v(\varnothing)=0.2, v(A)=0.4v(A)=0.4, v(B)=0.3v(B)=0.3, v(A,B)=0.7v(A,B)=0.7. A background distribution would instead average replacement evaluations. Conditional expectations define another possible missing-feature game; these choices need not agree.

The illustrated allocation is baseline 0.20.2, A credit 0.30.3, B credit 0.20.2. Horizontal lengths share a score scale, so the endpoint is exactly 0.70.7. Exact Shapley values satisfy this additive accounting for the specified game; numerical approximations can have residual error. The SHAP paper, sections 2–4 develops additive attributions and their defining properties.

Average Over Arrival Orders

A adds different amounts depending on which feature arrives first.
A adds different amounts depending on which feature arrives first.

There are two equally weighted arrival orders. If A arrives before B, its gain is 0.4−0.2=0.20.4-0.2=0.2. If B is present first, A adds 0.7−0.3=0.40.7-0.3=0.4. Thus ϕA=(0.2+0.4)/2=0.3\phi_A=(0.2+0.4)/2=0.3. The two arrow lengths use the same score scale; the second is twice the first. This averaging splits the interaction term between the two features.

Rank This Prediction's Credits

A receives more Shapley credit than B for this query and reference.
A receives more Shapley credit than B for this query and reference.

B adds 0.3−0.2=0.10.3-0.2=0.1 when first and 0.7−0.4=0.30.7-0.4=0.3 when second. Its mean gain is 0.20.2. The bars rank A above B by credit magnitude for this query and reference. They do not rank the features globally, establish fairness, or prescribe an intervention. The allocation accounts for 0.50.5 above baseline; it does not explain the baseline itself.

A Local Slope Can Change

Changing the held feature changes the local slope.
Changing the held feature changes the local slope.

The solid b=1b=1 slice has slope 0.40.4. The dashed b=0b=0 slice is 0.2+0.2a0.2+0.2a, with slope 0.20.2. Both use identical axes. The interaction makes sensitivity depend on the held value of B. Neither slope equals A's Shapley credit 0.30.3 here: a change per unit and an allocated score difference answer different questions. There is no general requirement that LIME coefficients equal SHAP credits.

Check What Masks Mean

Independent masks can query feature combinations absent from the data.
Independent masks can query feature combinations absent from the data.

Now add a support assumption: suppose observed feature pairs are only (0,0)(0,0) and (1,1)(1,1), shown as solid dots. The dashed diagonal is a guide, not extra observations. Independently replacing just one feature queries the crossed (1,0)(1,0) or (0,1)(0,1) combinations. Those evaluations are defined for our formula but unsupported by the assumed data.

A conditional game or grouped features changes the question being asked. Neither choice automatically reveals causality. Report the masking convention and check whether the queried combinations represent plausible inputs.

Change Only The Reference

A different reference changes the baseline and feature credits.
A different reference changes the baseline and feature credits.

Return to the original formula and query (1,1)(1,1), now using the single reference (0.5,0.5)(0.5,0.5). The four coalition scores become 0.4,0.55,0.5,0.70.4,0.55,0.5,0.7. A receives [(0.55−0.4)+(0.7−0.5)]/2=0.175[(0.55-0.4)+(0.7-0.5)]/2=0.175. B receives [(0.5−0.4)+(0.7−0.55)]/2=0.125[(0.5-0.4)+(0.7-0.55)]/2=0.125.

The lower bar still ends at 0.70.7: 0.4+0.175+0.125=0.70.4+0.175+0.125=0.7. Same prediction, different comparison. Both rows use 560 drawing units per score unit. A reference is part of the explanation, not an invisible default.

Predict A Fresh Query

A new query needs new feature credits even with the same reference.
A new query needs new feature credits even with the same reference.

For (1,0.5)(1,0.5) with reference (0,0)(0,0), first calculate the output: 0.2+0.2+0.05+0.1=0.550.2+0.2+0.05+0.1=0.55. The coalition scores are 0.2,0.4,0.25,0.550.2,0.4,0.25,0.55. A receives [(0.4−0.2)+(0.55−0.25)]/2=0.25[(0.4-0.2)+(0.55-0.25)]/2=0.25; B receives [(0.25−0.2)+(0.55−0.4)]/2=0.10[(0.25-0.2)+(0.55-0.4)]/2=0.10. The bar checks 0.2+0.25+0.10=0.550.2+0.25+0.10=0.55. Reusing the old credits would explain the wrong prediction.

Where It Breaks

Agreement on observed data does not identify behavior outside its support.
Agreement on observed data does not identify behavior outside its support.

For a separate boundary example, compare two candidate models hA(a,b)=0.2+0.5ah_A(a,b)=0.2+0.5a and hB(a,b)=0.2+0.5bh_B(a,b)=0.2+0.5b. If observations contain only (0,0)(0,0) and (1,1)(1,1), both models produce 0.20.2 and 0.70.7. A filled dot and open square coincide on each observed row.

At the unsupported probe (1,0)(1,0) they separate: hA=0.7h_A=0.7, hB=0.2h_B=0.2. Observed agreement cannot determine which extension is appropriate. Attributions to a chosen model can faithfully describe that model while still failing to justify a real-world intervention. Evaluate the model, the reference and perturbation assumptions alongside the explanation.

The Quick Version

  • A score alone does not allocate feature credit.
  • LIME fits an interpretable model under a chosen neighborhood.
  • SHAP credits depend on the missing-feature game and reference.
  • Arrival-order averaging distributes interactions across features.
  • Local slopes and allocated score differences are different quantities.
  • Correlated-feature masks can leave observed support.
  • New queries need new credits.
  • Additive accounting does not establish causal recourse or fairness.