Skip to content
AI360XpertBeta

Permutation Feature Importance

Shuffle one column of held-out data and watch the score fall. A big drop means the model leaned on that feature, and almost no drop means it did not.

Shuffling Age drops accuracy from 0.92 to 0.71 for a 0.21 importance score, while shuffling Income drops it only to 0.89, ranking Age far above Income
Shuffling Age drops accuracy from 0.92 to 0.71 for a 0.21 importance score, while shuffling Income drops it only to 0.89, ranking Age far above Income

Why Does This Exist?

A trained model on twenty columns never tells you which columns carried the decision. Built-in scores such as tree split counts exist, but they favor high-cardinality columns like zip code or user id, which offer many cheap split points, and they do not exist at all for neural networks.

Two ideas make the rest click. A validation set is data the model never trained on, so scores on it measure generalization rather than memory. Accuracy is the fraction of those rows the model gets right. If you have not ranked raw drivers yet, start with feature importance.

This page covers the sabotage test: break one column on held-out data and measure the fall. The curve that shows how predictions move as one feature varies lives in partial dependence and ICE. The reason the test must run on unseen rows lives in overfitting and underfitting.

Think of It Like This

The band with one randomized player

A band plays a song well and you wonder who carries it. You run one experiment per player: everyone plays normally except the bassist, who must play random notes. The song still sounds fine, so the bass was not load-bearing that night.

Next you randomize the lead singer while the rest play normally, and the song collapses. Nothing about the singer changed except the notes coming out, so the size of the collapse is the measure of reliance. Each musician is one feature, the normal performance is the baseline score, random notes are the shuffled column, and the drop in quality is the importance.

Where it stops: musicians hear each other and adjust, covering for the randomized player. Shuffled columns get no such cover, and two players covering the same part split the credit, which is exactly the correlated-features trap below.

How It Actually Works

Call the held-out score SS, meaning accuracy on validation rows. Call SjS_j the accuracy after shuffling column jj, meaning every other column stays fixed while column jj is randomly reordered across rows. Shuffling destroys the link between column jj and the target while keeping the column's own distribution intact. Importance is the plain drop:

Ij=S−SjI_j = S - S_j

A large IjI_j means the model leaned on column jj. A value near zero means the column could be noise, or its twin carries the same signal, which the first pitfall below separates.

Worked example: Age against Income on 500 rows

A model scores 460 of 500 validation rows correctly, so the baseline is:

S=460/500=0.92S = 460 / 500 = 0.92

Shuffle Age across the 500 rows and re-score: 355 correct, so:

Sage=355/500=0.71Iage=0.92−0.71=0.21S_{\mathrm{age}} = 355 / 500 = 0.71 \quad I_{\mathrm{age}} = 0.92 - 0.71 = 0.21

Restore Age, shuffle Income instead: 445 correct, so:

Sincome=445/500=0.89Iincome=0.92−0.89=0.03S_{\mathrm{income}} = 445 / 500 = 0.89 \quad I_{\mathrm{income}} = 0.92 - 0.89 = 0.03

Age outranks Income seven to one, 0.21 against 0.03. Repeat once per column, restore each column before shuffling the next, and sort the drops. Run the shuffle a few times with different random orders and average, because one shuffle is one noisy draw. For per-prediction shares rather than this global ranking, see SHAP.

Why the distribution is preserved

Shuffling reuses the column's own values, so the model never sees an impossible age like 400. The test breaks only the pairing between that column and the target. Replacing values with zeros or means would instead feed the model inputs it never trained on, mixing reliance with strangeness.

Code

The point is the check: rebuild the worked example drops from raw counts.

baseline_correct: int = 460age_correct: int = 355income_correct: int = 445rows: int = 500
baseline: float = baseline_correct / rowsage_drop: float = baseline - age_correct / rowsincome_drop: float = baseline - income_correct / rows
print(f"{baseline:.2f} {age_drop:.2f} {income_drop:.2f}")  # -> 0.92 0.21 0.03

The printout matches the hand computation: baseline 0.92, Age importance 0.21, Income importance 0.03. Change age_correct to 445 and the Age drop falls to 0.03, tied with Income, which is how the ranking responds.

Watch Out For

Correlated twins split the credit

Two columns carry the same signal, such as Age and Birth Year. Shuffling Age barely moves the score because the model leans on Birth Year for those rows, and shuffling Birth Year likewise barely moves it. Both report near zero and look useless, while dropping both together would collapse the score.

Symptom: two known-related columns both rank near the bottom while domain sense says the signal matters. Fix: check correlations before ranking, group twins and shuffle the group together, or keep one representative and re-run.

Running the test on training rows

A model that memorized noise, such as user id, scores well on training rows using exactly that column. Shuffling it on training data reports a huge drop, crowning a column that helps nowhere else.

Symptom: junk identifiers top the ranking while held-out accuracy lags far behind training accuracy. Fix: run every shuffle on validation or test rows only, never on the training set.

The Quick Version

  • Permutation importance shuffles one column of held-out data and reports the score drop as that feature's importance.
  • Our model drops from 0.92 to 0.71 when Age is shuffled, so Age scores 0.21, while Income scores only 0.03.
  • Shuffling keeps the column's own values and breaks only its pairing with the target, so inputs stay realistic.
  • The method works on any model because it treats the model as a black box and only re-scores it.
  • Correlated columns split the credit and each reads too low, so group twins before ranking.
  • Always shuffle validation rows, since training rows let memorized noise top the ranking.