Noise, Outliers and Anomaly Detection

The same statistics, the opposite action. A height of 3.2 metres is a typing mistake you delete; an odd transaction is the fraud you were hired to catch.

Age 34 looks ordinary on its own and 40 years of driving looks ordinary on its own, yet together they describe someone who started driving at minus six, which no column-by-column screen can see.

Why Does This Exist?

Two rows in a hospital table. One records a height of 3.2 metres. One records a resting heart rate of 210.

Both sit far out in the tail. Both trip any threshold you write. And the correct handling is opposite: the height is a decimal point in the wrong place and should go, while the heart rate may be the very event the model exists to predict. Delete the second and you have removed your positive class.

Detection tells you a value is unusual. It cannot tell you whether unusual means broken. Fraud, equipment failure, disease onset, intrusion, a demand spike: each is an outlier you must keep, and some of them are the label. So the first question is about provenance, never statistics: could this measurement physically have happened?

Three words get swapped and should not be. Noise is small random error around a true value, present in every measurement, and you model through it. An outlier is a point far from the rest, error or real. An anomaly is a point you went looking for, because finding it is the job.

This page is the detection-and-handling companion to the visual treatment in anomaly detection, which shows the same hunt beat by beat. Its neighbours on method are descriptive statistics for the summaries these rules are built from, and feature scaling because every distance-based detector needs scaled columns first.

The one idea you need first: a percentile is a value below which a given share of the data falls, so the 25th percentile has a quarter of the points beneath it. Percentiles barely move when one value goes wild, and that stubbornness is the whole reason the robust rules below beat the famous one. Means and standard deviations, by contrast, let one wild value drag them around.

Think of It Like This

A flatline on a hospital monitor

A bedside monitor goes flat and the alarm sounds. Two things make that trace, and they are as far apart as two things get.

An electrode came unstuck. The patient is fine, sitting up, mildly annoyed by the noise, and the fix is to reattach the lead. Or the heart has stopped, the trace is accurate, and every second spent doubting the equipment is a second lost.

The trace is identical. Nurses are trained for exactly this moment: look at the patient, not the machine. What separates the two cases is never inside the signal. It lives in the world the signal came from.

That is your outlier. The statistic says the value is extreme. Whether it is a loose lead or a cardiac arrest comes from knowing how the number was produced, which no formula on the column can supply.

How It Actually Works

Detection splits on one axis that matters more than the choice of algorithm: are you screening one column at a time, or scoring the row as a whole?

One column at a time

The z-score measures distance from the mean in standard deviations, and the 3-sigma rule flags anything past three. Tukey IQR rule flags anything outside $[Q_1 - 1.5 \cdot \mathrm{IQR}, Q_3 + 1.5 \cdot \mathrm{IQR}]$ , where $Q_1$ and $Q_3$ are the 25th and 75th percentiles and $\mathrm{IQR}$ is the gap between them. The modified z-score swaps in order statistics throughout:

z = \frac{x - \bar{x}}{s} \qquad\text{versus}\qquad z_{\mathrm{mod}} = \frac{0.6745\,(x - \tilde{x})}{\mathrm{MAD}}

On the left, $\bar{x}$ is the mean and $s$ the standard deviation. On the right, $\tilde{x}$ is the median and $\mathrm{MAD}$ is the median absolute deviation: the median of how far each point sits from the median. The 0.6745 makes both scores agree on normal data, so the usual cutoff sits near 3.5.

The left rule has a defect its popularity hides. The mean and the standard deviation are computed including the suspect point, so a large outlier inflates both and widens the very bar it is tested against. Past a certain size it hides itself entirely. That is called masking, and with a few extreme values a 3-sigma screen can return nothing at all. The IQR rule and the modified z-score are built from percentiles, and one wild value moves a percentile by at most one position, which is why percentile-based rules are the defaults.

The whole row at once

A row can be unremarkable in every column and impossible in combination. Age 34 is ordinary. Forty years behind the wheel is ordinary. Together they describe someone who started driving at minus six, as the diagram shows. No column screen sees that, because no column holds anything strange.

Four methods score the row instead:

Mahalanobis distance measures distance from the centre after accounting for how columns vary together, so "far" stretches along the directions the data actually spreads in. Assumes one elliptical cloud, which real data often is not.
Isolation forest cuts the data with random splits and counts how few it takes to isolate each point. Odd points fall out early. Fast, scales well, handles many columns, and the usual first reach.
Local outlier factor compares a point local density with its neighbours, so it catches a point that is odd relative to its own cluster when densities vary across clusters.
One-class SVM learns a boundary around the normal region. Sensitive to its kernel settings and to the contamination rate you assume.

All four use distance, so all four need scaled columns first. Run isolation forest on raw income beside raw age and income decides everything alone.

What to do about it, in order

Deleting is last, never first.

Investigate. Pull the raw record. Ten minutes here often surfaces a unit change or a broken sensor, fixing a whole class of rows rather than one.
Fix it when the true value is recoverable: a misplaced decimal, kilograms logged as grams, a timezone slip.
Cap it. Winsorising pulls extremes back to a percentile, usually the 1st and 99th. The row and its direction survive; the magnitude stops dominating.
Transform it. A log compresses a long right tail so extremes stop dominating, without discarding anything. See non-linear transformations.
Change the model. Trees barely notice extreme features, since a split only cares about order. For linear models a Huber loss is quadratic near zero and linear in the tails, so one bad point pulls far less than under squared error.
Drop it, with the reason recorded. Fine when you can name a physical cause. Never fine as a way to move a metric.

Worked example

Ten values: 10, 11, 12, 12, 13, 13, 14, 14, 15, 66. They sum to 180, so the mean $\bar{x}$ is 18.0, and the standard deviation $s$ is 16.062, most of it contributed by the 66 alone. The 3-sigma ceiling is therefore

18.0 + 3 \times 16.062 = 66.187

and the value under test is 66. Nothing gets flagged. Its z-score is $(66 - 18.0)/16.062 = 2.99$ , just under the line, because the point pushed the line out past itself.

Now the percentile route. Sorted, the 25th percentile is 12 and the 75th is 14, so the IQR is 2 and the upper fence is $14 + 1.5 \times 2 = 17$ . The 66 clears that by nearly four times. The modified z-score is starker: the median is 13, the absolute deviations are mostly 0, 1, or 2, their median (the MAD) is 1, giving $0.6745 \times (66 - 13)/1 = 35.7$ against a cutoff near 3.5. Same ten numbers. One rule says 2.99 and shrugs; the other says 35.7.

Code

1import numpy as np2
3x = np.array([10, 11, 12, 12, 13, 13, 14, 14, 15, 66], dtype=float)4med = float(np.median(x))5
6def iqr_upper(v: np.ndarray) -> float:7    """Percentile-built fence: one extreme value barely moves it."""8    q1, q3 = np.percentile(v, [25, 75])9    return float(q3 + 1.5 * (q3 - q1))10
11mad = float(np.median(np.abs(x - med)))          # median absolute deviation12print(x.mean(), round(float(x.std()), 3))        # -> 18.0 16.06213print(round(float(x.mean() + 3 * x.std()), 3))   # -> 66.187  nothing flagged14print(iqr_upper(x))                              # -> 17.0  the 66 clears it15print(med, mad)                                  # -> 13.0 1.016print(round(0.6745 * (x.max() - med) / mad, 1))  # -> 35.7  modified z-score

The gap between 66.187 and 17.0 is the entire argument for percentile-based rules, computed from the same ten numbers both rules saw.

Watch Out For

Dropping outliers because the metric improves

RMSE falls, the residual plot tidies, and it feels like cleaning. Check what left the building.

On a claims model the expensive claims are the tail. On churn, the heavy users. On a demand forecast, the spike you were asked to plan capacity for. Removing them makes the model excellent at the middle of the distribution, which is the part nobody needed a model for, and the error you get charged for lives in the part you deleted.

A quieter second effect: a scaler refitted on the trimmed column has a smaller spread, so in production, where the tail arrives whether you like it or not, those rows scale to values the model never saw. You did not remove the tail. You removed your ability to handle it. Before any deletion, name a physical cause and confirm the tail is not what you are graded on. Two noes means cap, transform, or change the loss.

Screening every column independently and calling it cleaning

for col in df.columns: df = df[iqr_mask(df[col])] looks thorough. It is a row shredder.

Each column removes its own small share, and the survivors are the intersection. With 20 columns each flagging about 1 percent, the surviving fraction is $0.99^{20} = 0.818$ , so 18 percent of rows go for nothing in particular, since almost every deleted row was flagged by one column and fine in the other 19. At 100 columns you keep 37 percent of your data.

The deletion is not neutral either. Rows extreme on one dimension tend to be extreme on a correlated one, so you remove a structured slice rather than a random sample, which is the same population shift missing data warns about through a different door. Score the row instead: one isolation forest over scaled columns gives a single number per row you can rank, inspect, and threshold once, starting with the top 50.

The Quick Version

Ask whether the value is a broken measurement or the rare event you are paid to catch. Detection cannot answer that; provenance can.
Noise is random error you model through. An outlier is far from the rest. An anomaly is the thing you went hunting for.
3-sigma uses the mean and spread the outlier itself inflated, so large outliers mask themselves. The IQR rule and the modified z-score come from percentiles and do not.
On ten values ending in 66, the 3-sigma ceiling is 66.187 and flags nothing. The IQR fence is 17.0 and the modified z-score is 35.7.
A row can be ordinary in every column and impossible in combination: age 34 with 40 years of driving started at minus six.
For whole-row scoring use Mahalanobis, isolation forest, local outlier factor, or a one-class SVM, always over scaled columns.
Act in order: investigate, fix, cap, transform, change the model or loss, then drop with a recorded reason.
Deleting to move a metric removes the tail you are graded on and quietly refits every scaler downstream.
Column-by-column filtering compounds: 20 columns at 1 percent each costs 18 percent of rows. Score rows instead.