Model Explainability (XAI)
A denied loan needs a reason a person can act on. Explainability splits one prediction into per feature credits so income, debt and age each own a signed share.
Why Does This Exist?
A linear score is readable: each extra square foot adds a fixed amount. A deep net or a large tree ensemble is not. Hundreds of nonlinear interactions decide, and the weights say nothing a loan officer can repeat.
For movie picks that opacity is fine. For a denied mortgage it is not lawful or decent to answer with hidden layer patterns. The bank must say which factors moved this decision and what the applicant could change. SHAP gives exact additive shares for that audit. LIME fits a local line for fast debugging. Permutation importance ranks global reliance by shuffling. This page covers when each lens fits and what none of them proves.
Think of It Like This
A silent chef with kitchen cameras
A gifted chef takes 50 ingredients and serves one soup without a word. When it lands too salty you cannot ask what caused it.
Cameras plus small tests help: hide the soy sauce once and taste the change, weigh pours from footage, rank bottles by effect. Ingredients map to features, the final taste maps to the prediction, hiding one bottle maps to masking a feature, and the ranked pour list maps to attribution shares.
Where it stops: cameras record what the chef did, not what salt does to health. Attribution explains the recipe, not the world. A top ranked bottle may only mark a correlated habit, and removing it from the shelf need not change the diner.
How It Actually Works
Call global what holds on average across rows. Call local what moved this one row. Call intrinsic a small model that is readable by construction. Call post hoc a second algorithm that probes a frozen black box. Almost all modern practice is post hoc because accuracy demands the box.
Global tools answer manager questions. Shuffling one column and measuring the score drop says how much the model leans on it. Local tools answer applicant questions. Sampling neighbors, weighting the near ones highest, and fitting a short line says what pushed this denial. Exact shares answer auditor questions. Game theory splits the gap between base rate and prediction into credits that sum exactly, so no share contradicts another.
Worked example: one denial that sums exactly
Base approval rate is 0.40. The model scores this applicant 0.60. The gap is 0.20 to explain. Attribution returns income plus 0.25, debt minus 0.10, age plus 0.05:
Check the sum stepwise:
Income tops the list, debt is the drag the applicant can address, age is small. The same applicant through a local line gets a slope that reads as income effect nearby, which should agree near this point and may diverge farther away. That agreement check is the diagram: one bar from 0.40 to 0.60 with three signed segments in order.
Why additive matters
When shares sum exactly, removing the top two and re-scoring has a predictable meaning, and two similar applicants cannot receive contradictory stories without showing in the numbers. Non additive weights can both look plausible and fail to reconstruct the score, which is why audits demand the exact split.
Code
The point is the audit check: rebuild the 0.60 prediction from its signed shares and flag any gap.
def audit_split(base: float, credits: list[float], scored: float, tol: float = 1e-9) -> str: """Rebuild a prediction from additive credits and report the gap.""" rebuilt: float = base + sum(credits) gap: float = abs(rebuilt - scored) return f"rebuilt {rebuilt:.2f} gap {gap:.2f} ok {str(gap <= tol)}"
print(audit_split(0.40, [0.25, -0.10, 0.05], 0.60)) # -> rebuilt 0.60 gap 0.00 ok Trueprint(audit_split(0.40, [0.25, -0.10], 0.60)) # -> rebuilt 0.55 gap 0.05 ok FalseThe printout matches the hand sum: full shares rebuild 0.60 with zero gap, while dropping age rebuilds 0.55 with a 0.05 gap and fails. Change debt from minus 0.10 to minus 0.30 and watch the rebuilt score fall to 0.40, which names the intervention in numbers.
Watch Out For
Reading attribution as causation
A global rank says hospital visits top mortality risk. A team treats visits as the cause and proposes fewer visits, while visits only mark sicker patients.
Symptom: an intervention on the top feature moves nothing, or a renamed proxy takes its place next run. Fix: read shares as what the model leaned on, then test the change with an experiment or a causal design before shipping, and keep the audit to model logic rather than world claims.
Trusting a local line where features move together
Income and zip correlate at 0.9. Masking income invents an executive with no pay history, an applicant outside training, and the local line swings run to run while exact shares stay consistent.
Symptom: nearby explanations disagree, neighbor weights collapse toward 0.14 far away, and masked samples look impossible. Fix: distrust lonely local lines under heavy correlation, compare against exact shares and permutation importance, and say the limit plainly instead of shipping the prettiest line.
The Quick Version
- Black boxes in high stakes choices need reasons a person can act on, not hidden layer patterns.
- Global ranks say what the model leans on average, local lines say what moved one row, exact shares sum to the score.
- Our denial rebuilds as base 0.40 plus 0.25 minus 0.10 plus 0.05, which lands exactly on 0.60.
- Income tops the list at plus 0.25 and debt drags at minus 0.10, which directs the next action.
- Shares explain the recipe, never the world, and correlated masks invent people who do not exist.