Feature Scaling

Scaling only matters when the algorithm compares columns to each other. If it only compares a column against a threshold, the numbers can stay as they are.

Squared distances before and after standardising: unscaled, the nearest neighbour is decided by income alone, and standardised, the same two candidates flip winner.

Why Does This Exist?

Two columns: annual_income in the tens of thousands, years_experience in single digits. Run K-nearest-neighbours and the distance between two people is, to three decimals, the difference in their incomes. Experience contributes nothing.

Nobody chose that. It fell out of the units.

Here is the rule that replaces the memorised list. Scaling matters exactly when the algorithm computes a distance, a dot product, or a penalty across columns. All three add contributions from different columns into one number, and a sum only means something when its terms are comparable. If the algorithm only compares a column against a threshold, scaling changes nothing.

Think of It Like This

Rope quoted in metres, rope quoted in centimetres

You need 40 metres of rope and two suppliers quote you numbers. One says 38, meaning metres. The other says 4,100, meaning centimetres. You add the numbers as written and buy from the second supplier, because 4,100 dwarfs 38.

Nobody mismeasured. You compared two numbers that were never on the same ruler. Divide each quote by its own unit first and the comparison becomes honest: 38 metres against 41.

Where it stops: metres and centimetres convert exactly, and neither supplier is wrong about the length. Income and experience have no true common unit to convert into. Scaling does not discover the fair comparison the way unit conversion does. It imposes one, and different impositions give different answers, which is exactly what the diagram above shows.

How It Actually Works

Apply the rule and the list falls out.

Needs scaling:

KNN and K-means, because both are literally a distance.
SVM with an RBF kernel, where the kernel is a distance and gamma is one width for all of it.
PCA, which maximises variance, so the biggest raw variance wins the first component.
Any L1- or L2-penalised linear model, because the penalty sums coefficients across columns, and small units mean a big coefficient to punish.
Every neural network trained by gradient descent, because unscaled inputs stretch the loss surface into a thin ravine where one step size cannot serve both directions.

Does not need scaling: trees, random forests, gradient boosting. A tree asks x > 3. After standardisation it asks x > -0.42, and both send exactly the same rows left and right. Not harmful, just pointless.

The three methods

Standardisation subtracts the mean and divides by the standard deviation:

z = \frac{x - \mu}{\sigma}

Here $\mu$ is the mean and $\sigma$ is the standard deviation. The output has mean 0 and standard deviation 1, and it is unbounded. This is the default.

Min-max squashes into a fixed interval, usually 0 to 1, as $(x - \min) / (\max - \min)$ . Use it when something downstream needs bounded input. Both parameters come from two rows, so one extreme value sets max and flattens everything else.

Robust scaling subtracts the median and divides by the interquartile range $Q_3 - Q_1$ , the width of the middle half. Both depend on rank, not magnitude, so extreme values barely move them. This is the answer when outliers are present.

Row scaling is a different thing

Unit-norm scaling divides each row by its own length. Column scaling makes features comparable to each other. Row scaling makes records comparable in direction and discards magnitude, which is what cosine similarity wants. Mixing the two up is a real bug with a quiet symptom: every record looks the same length, and nobody notices for weeks.

Sparse columns

A sparse matrix stores only its non-zeros. Subtract a mean and every stored zero becomes $-\mu$ , so a matrix that was 0.1% full becomes 100% full and stops fitting in memory. Use with_mean=False or MaxAbsScaler, which rescale without centering.

Worked example

The column [10, 12, 11, 13, 1000]. Four ordinary values, one outlier.

Min-max takes min 10 and max 1000, a range of 990: $(12 - 10) / 990 = 0.00202$ , $(11 - 10) / 990 = 0.00101$ , $(13 - 10) / 990 = 0.00303$ , with 10 at 0 and 1000 at 1. The four ordinary values share a window of about 0.003.

Robust scaling takes the median, 12, and the interquartile range. Sorted, the values are 10, 11, 12, 13, 1000, so $Q_1 = 11$ , $Q_3 = 13$ , and the IQR is 2. That gives $-1.0$ , $-0.5$ , $0$ , $0.5$ , and $(1000 - 12) / 2 = 494$ . The ordinary values get a window of 1.5 while the outlier stays visible at 494, available for a decision instead of silently setting the ruler.

Standardisation suffers too: its standard deviation of 395.4 comes almost entirely from the outlier, so the four ordinary values land at $-0.504$ , $-0.499$ , $-0.501$ and $-0.496$ , distinguished only in the third decimal.

Code

1import numpy as np2
3x: np.ndarray = np.array([10.0, 12.0, 11.0, 13.0, 1000.0])4
5def min_max(a: np.ndarray) -> np.ndarray:6    return (a - a.min()) / (a.max() - a.min())7
8def standardise(a: np.ndarray) -> np.ndarray:9    return (a - a.mean()) / a.std()10
11def robust_scale(a: np.ndarray) -> np.ndarray:12    q1, q3 = np.percentile(a, [25, 75])13    return (a - np.median(a)) / (q3 - q1)14
15print(np.round(min_max(x), 5).tolist())  # -> [0.0, 0.00202, 0.00101, 0.00303, 1.0]16print(np.round(standardise(x), 3).tolist())  # -> [-0.504, -0.499, -0.501, -0.496, 2.0]17print(np.round(robust_scale(x), 2).tolist())  # -> [-1.0, 0.0, -0.5, 0.5, 494.0]

Read the first and last lines together: min-max leaves the ordinary values in a window of about 0.003, robust scaling in one of 1.5.

Watch Out For

Fitting the scaler on everything, or refitting it at serving time

scaler.fit_transform(X) followed by train_test_split fits the mean and standard deviation on rows you are about to hold out, so the test set was standardised using knowledge of itself. The validation number improves and production does not. That is data leakage.

The stranger version happens at serving. Refit on each incoming batch and the same customer scores differently depending on who else arrived: alone they look average, alongside a whale they are an outlier. Nothing errors, and the scores are not reproducible even in principle.

Fit once, on training rows, and persist the parameters with the model artefact.

Scaling columns that should not be, or a target you forget to invert

StandardScaler on the whole frame hits your one-hot columns too. A binary indicator becomes something like -0.31 and 3.2, which is annoying for a linear model's interpretability and destroys the sparsity you were counting on. Select numeric columns by name with a ColumnTransformer.

Scaling the target is separate. Often right for a network, and it means your model no longer predicts prices. It predicts standardised prices. Skip the inverse transform and your metric is in units nobody recognises while your predictions are out by orders of magnitude.

The Quick Version

Scaling matters when the algorithm computes a distance, a dot product, or a penalty across columns. Otherwise it does not.
Needed for KNN, K-means, RBF SVM, PCA, regularised linear models, and gradient-descent-trained networks.
Irrelevant for trees and boosting: x > 3 and its scaled equivalent split identically.
Standardisation is the default, min-max when you need bounded output, robust scaling when outliers are present.
Min-max takes both parameters from two rows, so one extreme value flattens the rest into a sliver.
Unit-norm row scaling is different: records compared by direction, not columns aligned.
Never subtract a mean from a sparse matrix. Use with_mean=False or MaxAbs.
Fit on training rows, persist the parameters, never refit at serving.