LIME
You cannot explain a complex model globally, but zoomed in close on one prediction the boundary looks like a straight line, and LIME fits a simple model to that small patch.
Why Does This Exist?
Global tools answer global questions. Permutation importance says how much the model depends on age overall, and partial dependence shows the average effect. Neither answers the question a rejected applicant actually asks: why was my specific loan denied. The broader explainability landscape, including when a local reason is enough and when it is not, lives in model explainability.
A large neural network's decision boundary is a complicated high-dimensional curve with no readable equation. But calculus makes one promise: zoom in close enough on any smooth curve and it looks like a straight line. LIME, short for Local Interpretable Model-agnostic Explanations, turns that promise into a procedure. Instead of explaining the whole model, it samples a cloud of altered inputs around the one decision, asks the black box to score them, and fits a simple readable model to that small patch. The simple model's weights are the explanation. Its closest sibling, with exact additive guarantees LIME lacks, is SHAP.
Think of It Like This
A flat parking lot on a round planet
Describing the whole Earth needs spheres, mountains, and valleys. But stand one person in one parking lot and the ground under their feet is flat enough to map with a straight edge. A local sketch answers where this one puddle flows without modeling the planet.
The planet is the global boundary, the parking lot is the neighborhood around one prediction, and the straight edge is the surrogate line. Where it stops: the lot really is nearly flat, while a model boundary can zigzag even under a microscope. When the patch itself is jagged, the straight edge lies, and the fidelity check in the next section is how you catch it.
How It Actually Works
Call the input to explain , meaning the denied applicant aged 30. Call the black box , meaning the model that returns a denial score for any input. Call the kernel width , meaning how fast neighbor influence fades with distance.
Sample the neighborhood
LIME copies the applicant and tweaks features slightly to make fake neighbors: ages 28, 30, 32, and 35, with other features jittered the same way. The black box scores each: 0.58, 0.62, 0.66, and 0.72. Nearby alterations stay plausible while far ones drift into strangers, which is why distance weighting comes next.
Weight by closeness
Each neighbor at distance from the applicant gets weight , so close points count and distant ones fade. At distances 0, 0.5, 1.0, and 2.0 the weights are 1.0, 0.6412, 0.1690, and 0.0008. The last neighbor barely votes, which is exactly right: an 80-year-old stranger should not explain a 30-year-old's denial.
Fit the readable line
LIME fits a plain line through the weighted neighbors, where is age and is the predicted score. On the four scored neighbors the least-squares slope is score per year of age, and the intercept is . The line predicts 0.58, 0.62, 0.66, and 0.72, matching the black box on the patch. Read aloud: each extra year of age adds about 0.02 to the approval score near this applicant, so youth pushed this decision toward denial. The same idea drawn beat by beat lives in the SHAP and LIME visual.
Code
The weighted neighborhood and the least-squares line from the worked example, in plain standard-library Python.
xs = [28, 30, 32, 35]ys = [0.58, 0.62, 0.66, 0.72]
mx = sum(xs) / len(xs)my = sum(ys) / len(ys)slope = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / sum((x - mx) ** 2 for x in xs)intercept = my - slope * mxprint(f"{slope:.3f} {intercept:.3f}")# -> 0.020 0.020print([f"{intercept + slope * x:.2f}" for x in xs])# -> ['0.58', '0.62', '0.66', '0.72']The slope 0.020 with intercept 0.020 reproduces all four black-box scores, which is the fidelity check: a surrogate that cannot replay its own neighborhood is not an explanation.
Watch Out For
Different seeds giving different reasons
The neighborhood is randomly sampled, so two runs on the same applicant can rank the top features differently when the sample count is low.
Symptom: rerunning the explainer swaps the headline reason from debt to income with no model change. Fix: raise the sample count until the top weights stop moving across seeds, and report the seed alongside the explanation so it can be replayed.
A straight line fitted to a jagged patch
LIME assumes the boundary is nearly linear inside the neighborhood. On chaotic models the boundary zigzags even at microscopic scale, and the line fits noise.
Symptom: the surrogate replays its neighbors poorly, with large gaps between and on the very points in this section's check. Fix: run the replay check from the code section first. If it fails, shrink the kernel width or admit the decision has no simple local story instead of shipping a tidy false one.
The Quick Version
- LIME explains one prediction by sampling altered neighbors, weighting them 1.0 down to 0.0008 by distance, and fitting a readable line to the patch.
- Our four neighbors aged 28, 30, 32, and 35 score 0.58, 0.62, 0.66, and 0.72 from the black box.
- The fitted line has slope 0.02 per year with intercept 0.02, replaying all four scores exactly.
- LIME never opens the model, so it works on any black box, unlike methods that need gradients or trees.
- Random sampling makes reruns wobble, so high sample counts and a fixed seed keep the story stable.
- A jagged local boundary breaks the linearity bet, and the replay check catches it before anyone reads the weights aloud.