Feature Selection

Three families of method, and which family you pick matters much less than whether the choosing happened inside the fold or on the whole dataset first.

Five columns take three routes: the filter keeps both identical twins, the wrapper searches subsets at high cost, and the embedded penalty keeps one twin and zeroes the other.

Why Does This Exist?

Forty-two columns go into the model. Nine do the work. The rest add variance, cost money at serving time, and give you thirty-three more pipelines that can break at three in the morning.

So you cut. Three families of method do it, they differ in cost and in what they can see, and choosing between them is what everyone argues about. It is also what matters least.

Here is the part that matters: selection is a fitted step. It reads your labels and decides from them, exactly like a target encoder. Run it once on the whole dataset and then cross-validate, and your score is measuring a choice that already saw the held-out rows. Which family you used stops mattering.

Think of It Like This

Packing for a trek with a 10 kilo limit

Five kits are piled in the gear room and you may carry two. First option: weigh every item on its own and take the two warmest per kilo. Fast, done in an afternoon, and you pack two identical fleeces and no first-aid kit, because a scale cannot see how items fit together.

Second option: walk a practice hill with different packs and keep the one that felt best. That answers the actual question, and it takes weeks. Try enough packs and some combination wins on a lucky day.

Third option: hand everything to a guide who plans the route and names the kit as part of it. Two slots get filled, everything that fits no slot stops being carried, and selection came free with a decision you were making anyway.

Then the part that decides whether any of it worked. Pick the kit after watching yourself summit and it always looks like the right kit. It cannot look otherwise.

How It Actually Works

The three families

Filter methods score each column against the target on its own: variance threshold, correlation, chi-squared, ANOVA F, mutual information. One pass, no model, cheap on 10,000 columns. Blind to exactly two things. Redundancy: two perfectly correlated columns both score highly, so you keep both. Interaction: a column that only helps alongside another scores zero.

Wrapper methods train the model on candidate subsets and compare. Forward selection adds a column at a time, backward elimination removes, recursive feature elimination refits and drops the weakest each round. Closest to the objective, and the most expensive: a forward search over 40 columns is hundreds of fits. It overfits its own search too, since you are picking among subsets by validation score (multiple comparisons).

Embedded methods get selection out of a fit you wanted anyway. An L1 penalty on the sum of absolute coefficients drives them to exactly zero rather than merely small, so lasso hands you a subset. Tree ensembles report split-based importances, gradient boosting reports total gain. One fit, no outer loop.

Importance is not causation

Impurity-based tree importance is biased toward high-cardinality and continuous columns. More distinct values means more split points to try, so more chances to look useful, and a hashed customer_id can land in your top five. Permutation importance is the honest version: shuffle a column in held-out data and see how far the score falls.

Correlated columns split their credit, so two near-duplicates each take about half and both look weak. None of it is causal either. High importance means the model leans on a column, which also happens when the column is a consequence of the target.

What fewer columns costs

Fewer columns means less variance, cheaper serving, fewer dependencies to monitor, and fewer chances of leakage through a column nobody reviewed. The cost gets skipped, though. Every removed column is one you stop collecting: instrumentation goes, the ETL step gets deleted, the vendor feed lapses. Wanting it back in six months means six months of history you do not have.

Worked example

Five rows, three columns. Column A is real signal, column B an exact copy, column N noise. The target is roughly twice the signal:

1A  B  N | y21  1  5 | 232  2  1 | 543  3  4 | 554  4  2 | 965  5  3 | 9

A correlation filter scores each column alone. With means 3, 3, 3 and 6, the absolute correlations are $|r| = 0.949$ , $0.949$ and $0.527$ , so it drops the noise and keeps both duplicates. It never compared the columns to each other: the numerator for A is $(-2)(-4) + (-1)(-1) + 0 + (1)(3) + (2)(3) = 18$ , and B repeats it exactly.

L1 with a penalty of 0.6 applies the soft threshold $w = \mathrm{sign}(\rho)\max(|\rho| - 0.6, 0)$ : $0.349$ , $0.349$ , $0$ . The noise goes to exactly zero while the signal survives shrunk. And in a joint fit the penalty keeps one twin and zeroes the other: once the first twin absorbs the signal, the second adds nothing worth its penalty. With exact ties, column order decides which twin survives.

Code

Plain standard library. The filter scores, then the L1 rule shrinks them.

1signal = [1.0, 2.0, 3.0, 4.0, 5.0]2target = [2.0, 5.0, 5.0, 9.0, 9.0]3noise = [5.0, 1.0, 4.0, 2.0, 3.0]4
5
6def pearson(a: list[float], b: list[float]) -> float:7    ma, mb = sum(a) / len(a), sum(b) / len(b)8    cov = sum((v - ma) * (w - mb) for v, w in zip(a, b)) / len(a)9    return cov / ((sum((v - ma) ** 2 for v in a) / len(a)) * (sum((w - mb) ** 2 for w in b) / len(b))) ** 0.510
11
12def soft_threshold(rho: float, lam: float) -> float:13    return (1.0 if rho > 0 else -1.0) * max(abs(rho) - lam, 0.0)14
15
16scores = [abs(pearson(c, target)) for c in (signal, signal, noise)]17print([round(s, 3) for s in scores])  # -> [0.949, 0.949, 0.527]18print([round(soft_threshold(s, 0.6), 3) for s in scores])  # -> [0.349, 0.349, 0.0]

max(abs(rho) - lam, 0.0) is the whole difference. Ridge shrinks a coefficient toward zero and leaves it non-zero forever. L1 clips it flat.

Watch Out For

Selecting on the whole dataset, then cross-validating

It reads innocently. SelectKBest(k=10).fit(X, y), then cross_val_score on the ten survivors. Two lines, each correct, wrong in that order.

The textbook demonstration settles it: 200 columns of pure standard-normal noise, 100 rows, coin-flip labels, so there is nothing to learn. Pick the top ten by ANOVA F across every row, then five-fold cross-validate a logistic regression on them, and it scores about 0.77. Move the selector inside the pipeline so each fold picks its own ten and it falls to about 0.48, the coin flip you started with.

No fold disagrees, because every fold's held-out rows helped choose the columns. The cross-validation is confirming a decision it cannot question. Selection goes inside the Pipeline, and tuning k needs its own inner loop (cross-validation schemes).

Trusting default tree importances and dropping a partner column

model.feature_importances_, sorted, cut at the top twenty. Fast, and wrong in two directions at once.

The ranking favours columns with many distinct values, so continuous ones float up while a predictive binary flag sinks. And correlated columns split their credit, so a pair carrying the signal shows up as two mediocre features.

The other half is interactions. is_premium can carry no signal alone and everything alongside discount_pct, so a filter scores it zero. Drop it and the surviving column stops working too.

Use permutation importance on held-out folds, group correlated columns before ranking, and confirm any removal by refitting without it. When the pool is too large for that, you have a generator problem (automated feature engineering).

The Quick Version

Three families. Filters score each column alone: fast, blind to redundancy and interaction. Wrappers train per subset: accurate, expensive, overfit their own search. Embedded gets selection free from the fit.
The family matters less than the placement. Selection is fitted, so it belongs inside the fold.
200 noise columns, 100 rows, random labels: selecting first scores about 0.77 cross-validated. Selecting inside the pipeline scores about 0.48.
Impurity importance is biased toward high-cardinality and continuous columns, and it is not causal. Permutation importance on held-out data is the honest version, and correlated columns split their credit so both look weak.
The filter keeps both duplicates at 0.949 each. L1 shrinks the signal to 0.349 and clips the noise to exactly 0, and in a joint fit column order decides which twin survives.
Fewer columns means less variance and cheaper serving. It also means you stop collecting the data, which is hard to undo.