Skip to content
AI360XpertBeta

Anomaly Detection

Most rows are ordinary and a few are strange in ways you cannot list in advance, so you score each row for strangeness and flag whatever passes a cutoff.

A z-score of 3.87 pushes the value 14 past the flag line at 3, while the normal value 12 sits at 1.81 and stays unflagged
A z-score of 3.87 pushes the value 14 past the flag line at 3, while the normal value 12 sits at 1.81 and stays unflagged

Why Does This Exist?

Most damaging rows never announce themselves. Fraud, intrusions, and failing sensors hide among thousands of ordinary records, and you cannot write down every bad pattern in advance because the next one will look different. A check that looks at one column at a time misses the real cases: each value looks fine on its own, and only the combination is wrong.

Here is the trap that makes this topic matter. A payments team reviews about 2,000 transactions a day, and roughly 2 percent turn out fraudulent. A model that predicts "not fraud" every single time, without even reading the data, scores 98 percent accuracy while catching zero fraud. Accuracy rewards silence here, so you need a discipline built around the rare rows and graded on them.

This page covers the frame every detector shares: score each row, draw a line, flag what crosses it, and grade the result on the rare class. Two concrete scorers get their own pages, Isolation Forest and one-class SVM. The full grading story lives in precision and recall. Single-column cleanup is a different job, covered in outlier detection, and the same metric trap shows up wherever classes are skewed, in imbalanced data. Two words you will need below: precision (of everything you flagged, how much was really bad) and recall (of everything really bad, how much you caught).

Think of It Like This

The night guard who learns the hum

A warehouse hires a night guard who first spends a week just listening. Freezers buzz, pipes tick, the roller door rattles when wind hits it. That hum is normal. On night eight a new sound cuts through, a sharp clang from the far corner, louder than anything in the first week. The guard walks over. That is the whole job: learn the usual, then investigate what breaks it.

Each sound gets a loudness in the guard's head, and there is a line past which feet move. A faint tick stays seated. A clang gets a torch. The line is the entire policy, and moving it changes the job: lower it and nights fill with false walks, raise it and a quiet break-in slips past.

Where it stops: a careful intruder tiptoes, making sounds the guard filed as normal, so no loudness line can catch mimicry. And the guard can point, never explain: flagging a clang is not naming the crime behind it.

How It Actually Works

Every detector is the same two-part deal. You compute an anomaly score for each row, written s(x)s(x), where xx means one row written as numbers and a larger score means stranger. Then you choose a threshold, written τ\tau and read "tau", which is the cutoff loudness past which feet move. The flag, written y^\hat{y} and read "y-hat", is 11 for investigate and 00 for leave alone:

y^={1if s(x)>τ0otherwise\hat{y} = \left\{ \begin{array}{ll} 1 & \text{if } s(x) > \tau \\ 0 & \text{otherwise} \end{array} \right.

The score is the modelling. The threshold is the policy. Confusing the two is where most bad deployments start.

How a row earns its score

Three families do the scoring, and they differ only in what "normal" means to them:

  • Distance from a centre. Fit a mean μ\mu (the centre) and a spread σ\sigma (the typical distance from it, read "sigma"), then score by how many spreads a row sits away. That count is the z-score. It is fast and honest in one dimension, and it trusts the centre you gave it.
  • Loneliness among neighbours. Score by how sparse or far a row's neighbourhood is. Normal rows sit in crowds, strange ones sit apart. No centre is assumed, which helps when normal has bends and clusters a single mean cannot describe.
  • What a model of normal cannot explain. Learn normal only, then flag what it reconstructs badly or walls off. Isolation Forest scores by how few random cuts isolate a row, and one-class SVM draws one boundary around normal and flags the outside. Both train without ever seeing a labelled attack.

Shapes come in three sizes. A point anomaly is one row wrong on its own. A contextual anomaly is wrong only given its setting, like a heating bill that is fine in January and wild in July. A collective anomaly is a group wrong together while each member looks fine alone, like logins that are each plausible until you see the sequence. Most default scorers hunt point anomalies; the other two need the context or the sequence built into the features first, or the scorer stays blind to them.

The shared assumption is that normal is common and strange is rare. When that breaks, when a third of the feed is attack, or normal drifts after a product change while the model still hums the old tune, scores go stale and flags follow. If you have plenty of labelled fraud and balanced classes, skip all of this and train a classifier instead: scoring without labels buys coverage of the unknown at the price of precision you could have had with labels.

How the cutoff turns scores into flags

Scores rank. The threshold decides, and it always charges you somewhere. Lower τ\tau and you catch more of the bad with more false walks; raise it and the walks stop while misses grow. That is the precision against recall trade, and precision and recall is where you settle it with numbers instead of vibes.

Pick the line from what you expect, not from what flatters. If about 2 percent of rows should trip, start near the 98th percentile of scores on clean reference data, then adjust against the cost of a miss versus a false walk. Never tune the line and report the result on the same rows: that loop always tells you the detector is brilliant. The same discipline, applied wherever rare classes meet headline metrics, is the subject of imbalanced data.

Worked example

Eight sensor readings from a healthy machine: 1010, 1111, 99, 1010, 1212, 1111, 1010, 99. Their sum is 8282, so the mean is μ=82/8=10.25\mu = 82 / 8 = 10.25. Squared distances from that centre sum to 7.57.5, so with the population spread the variance is 7.5/8=0.93757.5 / 8 = 0.9375 and σ=0.9375≈0.97\sigma = \sqrt{0.9375} \approx 0.97. The z-score says how many spreads a value sits from the centre:

z=x−μσwithμ=10.25,  σ≈0.97z = \frac{x - \mu}{\sigma} \quad \text{with} \quad \mu = 10.25, \; \sigma \approx 0.97

A new reading of 1414 lands far out:

z14=14−10.250.97≈3.87andz12=12−10.250.97≈1.81z_{14} = \frac{14 - 10.25}{0.97} \approx 3.87 \quad \text{and} \quad z_{12} = \frac{12 - 10.25}{0.97} \approx 1.81

At a flag line of τ=3.0\tau = 3.0, the 1414 at 3.873.87 trips the flag and the 1212 at 1.811.81 stays put. The band μ±3σ\mu \pm 3\sigma runs about 7.357.35 to 13.1513.15, which is the same decision drawn on the number line above: 1414 sits outside it, 1212 sits inside. Small shift, same verdict either way you compute it.

Code

The point is the check, not the formula: score the reference window, draw the line at 3.03.0, and watch the suspect trip it while a normal reading stays quiet.

import numpy as np

def z_scores(xs: np.ndarray) -> np.ndarray:    mu: float = float(np.mean(xs))    sigma: float = float(np.std(xs))  # population spread, ddof=0    return (xs - mu) / sigma

normal: np.ndarray = np.array([10.0, 11.0, 9.0, 10.0, 12.0, 11.0, 10.0, 9.0])tau: float = 3.0z_suspect: float = float((14.0 - float(np.mean(normal))) / float(np.std(normal)))z_ok: float = float((12.0 - float(np.mean(normal))) / float(np.std(normal)))
print(f"mean {np.mean(normal):.2f} spread {np.std(normal):.2f}")  # -> mean 10.25 spread 0.97print(f"suspect z {z_suspect:.2f} flagged {z_suspect > tau}")  # -> suspect z 3.87 flagged Trueprint(f"normal z {z_ok:.2f} flagged {z_ok > tau}")  # -> normal z 1.81 flagged False

The printout matches the hand computation above: centre 10.2510.25, spread 0.970.97, suspect 3.873.87 flagged, normal 1.811.81 left alone.

Watch Out For

Grading a rare-event detector with accuracy

The fastest way to make a useless detector look brilliant is to report accuracy where the event is rare. At 2 percent fraud the do-nothing baseline already reads 98 percent, so a stakeholder hearing "98.9 percent accurate" cannot tell whether the detector caught anything at all.

Symptom: a headline number near 100 percent paired with no mention of catches or false walks. Fix: report precision and recall on the rare class every time, and never accuracy alone when the base rate sits far from half and half.

Letting suspects set the normal

The centre and spread are fit from data, and anomalies pull them. Include the suspect week in the reference window and the mean slides toward it while the spread balloons, so the very row you hunt earns a smaller z than it should. Our 1414 would look tamer if it had been allowed to vote on μ\mu and σ\sigma.

Symptom: thresholds that need constant loosening, or worked examples that flag less than the math says they should. Fix: fit the centre and spread on a clean reference window, re-fit after removing flagged rows, or use a median-based centre that lone giants cannot drag.

The Quick Version

  • Anomaly detection scores each row for strangeness and flags what crosses a line you choose; the score is the model and the line is the policy.
  • A z-score turns "far" into a number: spreads from the centre, with 3.0 a common flag line and 3.87 a clear trip.
  • Each axis can look fine while the combination is wrong, so row-level scoring beats column-level checks for fraud, intrusion, and failure.
  • Moving the line trades false walks against misses, which is why precision and recall grade detectors and accuracy cannot.
  • Fit normal on clean data, because suspects that vote on the centre shrink their own scores and blind the detector.