Core ML

Feature Importance

The importance your tree library prints for free answers a narrower question than it looks like. Permuting a column on held-out data answers the one you meant.

Mean decrease in impurity credits a noisy high-cardinality column just for having more chances to win a split, while permutation importance measured on held-out data correctly reads it as worthless

Why Does This Exist?

A model ships, works, and the first question from anyone downstream is "why did it say that". Feature importance is the site's answer, and the honest version of that answer is more careful than a bar chart implies.

Here's the case we'll carry down the page. A bank's credit model scores 50,000 loan applications on twenty features: income, years employed, an internal risk score, a customer ID assigned at account opening, and a duplicate income field pulled from a second source that agrees with the first about 95% of the time. A regulator asks which features drive the decision, and whichever answer you print becomes part of a compliance filing.

Print the wrong kind of importance and you can end up defending a ranking where a meaningless ID column outranks income — not because the model is broken, but because the importance measure was answering a different question than the one asked.

Think of It Like This

Crediting a relay team by who was running when the gun went off

A four-person relay team wins, and a reporter wants to know who won it. Ask "who was holding the baton when they broke the tape" and you get one name — the anchor, always, because the race always ends on their leg. That's not nothing, but it isn't the answer to "who mattered", and it's a name the format of the race forced, not one the team's actual speed produced.

Ask instead "how much slower would this team have finished without this runner" — swap each one out for an average club runner, one at a time, and time the result. Now you learn something that would still be true if the race were run again with a different final leg. The first question measures where credit fell inside this one race. The second measures what each runner actually contributed.

Impurity-based importance is the first question. Permutation importance is the second.

How It Actually Works

MDI: cheap, built in, and answering a narrower question than it looks

Mean decrease in impurity, the importance every tree library prints for free, sums how much each feature reduced Gini impurity or entropy across every split it won, over every tree in the ensemble. It costs nothing extra — the number falls out of training itself — which is exactly why it ships as the default and gets treated as more authoritative than it is.

Two biases are built into that arithmetic, not bugs but consequences of what the sum is actually counting. A continuous or high-cardinality column offers far more candidate cut points than a binary flag, so it gets far more chances to win a split purely from having more places to try — importance inflated by opportunity, not by relationship to the target. And when two features carry the same information, as the bank's duplicate income fields do, split credit lands on whichever one the tree happened to draw first at each node, splitting a real signal into two moderate-looking scores instead of one strong one.

Permutation importance: measured on data the model never memorised

Permutation importance asks the relay question instead: shuffle one feature's values across the rows, breaking its relationship with the target while leaving every other feature untouched, and measure how much the model's score falls on data it never trained on. A feature the model genuinely depends on causes a real drop. A feature it never needed — the customer ID — causes almost none, because shuffling noise produces more noise.

Crucially this has to run on held-out data. Measure the drop on training data and a model that's overfit will appear to depend heavily on features it only memorised, the same failure cross-validation exists to catch elsewhere.

Where SHAP takes over

Permutation importance answers "how much does the model depend on this feature, overall". It can't answer "why did this applicant get declined" — that needs a per-prediction breakdown, splitting one row's score into an additive contribution from each feature, consistent with cooperative game theory's Shapley values. That's a heavier computation, worth its own page, and the tool to reach for once the question moves from ranking features in aggregate to explaining one decision.

Show Me the Code

A weak real signal beside a noisy, high-cardinality row-id column. MDI trains for free but reads the id as informative; permutation, run on held-out rows, calls it correctly.

1import numpy as np2from sklearn.ensemble import RandomForestClassifier3from sklearn.inspection import permutation_importance4
5rng = np.random.default_rng(3)6signal = rng.normal(size=4000)                     # a real but weak predictor7row_id = rng.integers(0, 4000, 4000).astype(float)  # unique-ish id column, pure noise8y = (signal + rng.normal(0.0, 3.0, 4000) > 0).astype(int)9
10x = np.column_stack([signal, row_id])11x_tr, x_te, y_tr, y_te = x[:3000], x[3000:], y[:3000], y[3000:]12
13forest = RandomForestClassifier(n_estimators=200, max_depth=4, random_state=0)14forest.fit(x_tr, y_tr)15perm = permutation_importance(forest, x_te, y_te, n_repeats=20, random_state=0)16
17print(f"MDI (training set):        signal={forest.feature_importances_[0]:.2f}  row_id={forest.feature_importances_[1]:.2f}")18print(f"permutation (held-out):    signal={perm.importances_mean[0]:.2f}  row_id={perm.importances_mean[1]:.2f}")19# -> MDI (training set):        signal=0.77  row_id=0.2320# -> permutation (held-out):    signal=0.09  row_id=0.00

MDI gives the noise column nearly a quarter of the credit, purely from having thousands of candidate cut points to try. Permutation on held-out rows drops it to zero, because shuffling a column that predicts nothing changes nothing.

Watch Out For

Reading importance as causation

High importance means the model leans on a feature for its predictions, and that leaning happens for reasons that have nothing to do with cause and effect: a feature can be a downstream consequence of the target, a proxy for something causal, or a coincidental correlate in this particular sample. A hospital-readmission model can rank "number of prior admissions" as its top feature — genuinely predictive, and not something a hospital can act on by simply admitting people less. Importance ranks predictive weight, not levers.

Reporting MDI rankings in a document someone will act on

A regulator, an executive, or an audit reads a feature-importance chart as a settled ranking of what matters, and MDI's cardinality bias and correlated-feature splitting mean the chart can be wrong in a specific, checkable direction — an ID column outranking income, or a real driver split across two near-duplicate columns and each looking moderate. Once a number leaves the training script and enters a report, use permutation importance on a held-out set, and note that two correlated features will each look weaker than the pair's combined effect.

The Quick Version

MDI sums how much impurity each feature's splits removed across an ensemble; it's free, but biased toward high-cardinality features and splits shared credit between correlated ones.
Permutation importance shuffles one feature at a time and measures the resulting drop in held-out performance, answering "how much does the model actually depend on this" rather than "how often did this feature win a split".
Run permutation importance on data the model never trained on; on training data it inherits the same overfitting a validation set exists to catch.
Neither method implies causation. High importance means the model leans on a feature, not that the feature drives the outcome.
SHAP values extend the same idea to individual predictions, splitting one row's score into per-feature contributions, once the question moves from "what matters overall" to "why did this row get this answer".