Feature Engineering

Why Does This Exist?

Two teams take the same forecasting problem. Same gradient-boosted trees, same rows, same tuning budget. One lands at 12% error and the other at 4%. Nothing about the model differed. One team handed it a raw timestamp column. The other handed it hours_since_previous_order, orders_in_prior_7_days and is_public_holiday.

That gap is feature engineering, and on tabular problems it is routinely wider than the gap between two algorithms.

The reason runs deeper than tidiness. A model is a function fitted to a fixed set of columns. It does not read your database, infer your business, or notice that two of your columns should be divided by each other. It sees a grid: one row per thing you want to predict about, one column per number you decided to show it. That grid is the design matrix, usually written $X$, and it is the model's entire universe. A pattern that is not expressed in $X$ cannot be learned from $X$.

So "which model should I use" comes after a bigger question: what does the model get to look at?

The vocabulary, in one paragraph. A feature is one input column: distance_km, is_weekend, word_count. A row is one thing you are predicting about. The target is what you are predicting. Feature engineering is everything that turns your records into that grid: making columns that did not exist, turning non-numbers into numbers, putting numbers on a comparable footing, and deleting what hurts.

Think of It Like This

How It Actually Works

Four operations, in this order

Each step assumes the previous one is done.

1. Construct. Create columns that do not exist yet. Ratios, differences, date parts, rolling aggregates, lags. This is where domain knowledge enters, and it has the most headroom.
2. Encode. Turn non-numbers into numbers. Categories, text, timestamps, identifiers. Every model consumes floats, so something has to do the conversion, and how you do it decides what the model can infer.
3. Scale. Put numeric columns on a comparable footing, and reshape distributions that break a model's assumptions.
4. Select. Remove columns that add noise, leak, cost money, or duplicate each other. Fewer features often means a better model, which surprises people.

Every one of those four steps learns from data

This matters more than any individual technique, and it is why this page points at leakage before anything else.

A scaler learns a mean and a standard deviation. An imputer learns a median. A one-hot encoder learns the list of categories. A target encoder learns a per-category average of the thing you are trying to predict. None of these are pure functions. Each is fitted, and what it gets fitted to is a choice you make.

Fit them on all your data and you have moved information out of your test set and into your training set. Your validation score improves. Production does not. That is data leakage, and preprocessing causes it far more often than anything exotic does.

The rule is short: fit on training rows, apply everywhere. Every fitted transform, every time, including inside every cross-validation fold.

What the model can and cannot work out for itself

Model families need different amounts of help, and knowing which is which saves a lot of pointless work.

• Linear models cannot discover interactions or curvature at all. If the effect is price / square_metres, you have to build that column. They also need scaled inputs, because a coefficient's size is entangled with its column's units.
• Trees and gradient boosting get monotone transforms for free. A split on x > 3 and a split on log(x) > 1.1 partition the rows identically, so scaling changes nothing. What they cannot do cheaply is arithmetic across columns. A ratio costs you one line and costs them a staircase of dozens of splits, approximated rather than exact.
• Neural networks learn features, given enough data and scaled inputs. That is why feature engineering feels dead in vision and text, where the encoder does it, and very much alive on tabular data, where you rarely have the volume to earn it.

Feature engineering did not die with deep learning. On raw modalities it moved into curation and representation, which is what most of the rest of this band is about.

Worked example

Predicting a health outcome from weight_kg and height_m. Take two people: 95kg at 1.62m, and 61kg at 1.80m.

On the raw columns they are awkward to separate, because the heavier person is also the shorter one. Any rule using weight alone gets the light-and-tall case wrong. Body mass index is $\text{weight} / \text{height}^2$:

One column, one line of code, and a single threshold now separates them cleanly. A gradient-boosted tree given only weight and height would need many axis-aligned splits to trace that curve, and it would still only approximate it. You handed it the exact answer instead.

That is the shape of the whole discipline: you know something about the world that the loss function cannot discover from the columns in front of it.

Code

Two habits in muscle memory: constructing a column, and fitting a transform on training rows only.

1import numpy as np2
3weight_kg = np.array([95.0, 61.0, 88.0, 54.0])4height_m = np.array([1.62, 1.80, 1.91, 1.55])5
6
7def build_bmi(weight: np.ndarray, height: np.ndarray) -> np.ndarray:8    return weight / height**29
10
11def fit_scaler(train: np.ndarray) -> tuple[float, float]:12    return float(train.mean()), float(train.std())13
14
15bmi = build_bmi(weight_kg, height_m)16print(np.round(bmi, 1))  # -> [36.2 18.8 24.1 22.5]17mean, std = fit_scaler(bmi[:2])18print(np.round((bmi - mean) / std, 2))  # -> [ 1.   -1.   -0.39 -0.58]

fit_scaler receives bmi[:2] and nothing else. That deliberate narrowing is the difference between a validation number you can trust and one you cannot, and it is why scikit-learn's Pipeline exists: it makes the narrowing structural instead of something you have to remember.

Watch Out For

The Quick Version

• The model only ever sees the design matrix you built. A pattern not expressed in the columns cannot be learned from them.
• Four operations, in order: construct, encode, scale, select.
• On tabular problems, the gap between good and bad features is usually wider than the gap between two algorithms.
• All four steps are fitted, each learns a statistic, so all four can leak. Fit on training rows, apply everywhere.
• Trees ignore monotone scaling but cannot do arithmetic across columns. Linear models cannot find interactions. Neural nets learn features, given volume.
• One good column replaces dozens of splits: $\text{weight} / \text{height}^2$ instead of a staircase.
• Fewer features frequently beats more.
• For every column, ask whether its value was knowable at prediction time. If not, it is a leak wearing a useful name.

What to Read Next

• Exploratory Data Analysis
• Feature Scaling
• Feature Selection