Core ML

K-Nearest Neighbors

No training at all: store every row, then answer each query by finding the k closest rows and taking a vote. The cost moved to prediction, it did not vanish.

The same query point gets opposite answers at k equals five and k equals eleven, because widening the neighbourhood changes who is allowed to vote

Why Does This Exist?

A plant has 800 pumps with maintenance history. For each one: vibration at the bearing housing in millimetres per second, motor speed in rpm, oil temperature, and whether it needed a bearing replaced that month. Tomorrow a pump gets measured and someone decides whether to pull it.

Every other model here assumes a shape — a line, a hyperplane, a tree — and fits parameters to it. This one refuses. Store the 800 rows; when a new pump arrives, find the rows most like it and see what happened.

That's the whole algorithm. "Training" is a memory copy — the work didn't disappear, it moved to prediction, where you pay it once per query, forever.

One thing to fix first. Write a pump's measurements as a list of numbers and it becomes a point in a space with one axis per measurement. Distance is the ordinary geometric gap between two such points.

Think of It Like This

The locksmith's board of blanks

A locksmith keeps a board of several hundred key blanks, each tagged with the lock it fits. You hand him a key. He doesn't measure it and derive a theory of keys — he holds it against the board, picks the three or four that look closest, and reads their tags. His accuracy comes from the board being large and well labelled.

Now hang the board badly. Position across it is length in millimetres, spanning 40; position down it is tooth depth in tenths of a millimetre, spanning 3. Every key of the same length now looks adjacent however differently it's cut, and he'll hand you the wrong answer with total confidence. His method didn't break. The board lied about what "close" means.

How It Actually Works

The distance is the model

Pick a distance and you've picked what "similar" means. Four worth knowing.

Euclidean is straight-line distance: square the differences, sum, take the root. It assumes every axis is comparable and that one unit means the same on each.

Manhattan sums absolute differences instead of squaring, so no single wild coordinate dominates. Steadier when a few features carry outliers you can't clean.

Minkowski generalises both with an exponent $p$ : at $p = 1$ it's Manhattan, at $p = 2$ Euclidean.

Cosine measures the angle and ignores length, so a tweet and an essay with the same word mix score as identical. That's what text wants, since length isn't the signal. Wrong for pumps, where magnitude is the signal. Full menu in distance metrics.

Scale it, or you have a one-column model

This ruins more deployments than everything else here combined.

Motor speed spans 2,200 rpm. Vibration spans 7.5 mm/s. So two pumps 300 rpm apart sit 300 apart, while two at opposite ends of the vibration range sit 7.5 apart — squared, tens of thousands against single digits.

Every neighbour now gets chosen by motor speed alone and the other columns are decoration. The model isn't weighting features badly; it's weighting them by your arbitrary decision to record speed in rpm rather than thousands of rpm.

Standardise every column, or min-max it into a fixed range, and all features arrive on comparable footing — the only assumption Euclidean distance ever made. Feature scaling covers which transform suits which column.

Choosing k, and the bill at prediction time

Small k follows the data closely. At $k = 1$ every training point owns a patch of space, so one mislabelled pump gets its own island of wrong predictions. Large k averages over a wide neighbourhood and, at the limit, predicts the majority class everywhere.

There's no formula. Sweep k on validation data and take the flat part of the curve, not the single best point, which is partly luck. Keep k odd for two classes so a vote can't tie. The diagram shows the stakes: one dataset, opposite answers at 5 and at 11.

Then the bill. Every query compares against every stored row, so cost is linear in training-set size and the whole set stays in memory. Spatial indexes help in low dimensions and degrade to brute force in high ones.

Which brings up the failure with no fix. Take 1,000 points spread uniformly at random and ask how much farther the farthest is than the nearest. In two dimensions, about a hundred times. At 100 features, 1.4 times. At 1,000 features, 1.1. Every point sits roughly the same distance from every other, so "nearest" stops carrying information. That's the curse of dimensionality, and here it's the end of the road rather than an inconvenience.

Show Me the Code

Same pumps, same k. The only difference is whether the columns were scaled first.

1import numpy as np2
3def vote(train: np.ndarray, labels: np.ndarray, queries: np.ndarray, k: int) -> np.ndarray:4    d2 = ((queries[:, None, :] - train[None, :, :]) ** 2).sum(axis=2)5    near = np.argsort(d2, axis=1)[:, :k]             # no fitting happened; the work is all here6    return (labels[near].mean(axis=1) > 0.5).astype(float)7
8rng = np.random.default_rng(5)9rpm = rng.uniform(700.0, 2900.0, 800)                # motor speed, in the thousands10vib = rng.uniform(0.5, 8.0, 800)                     # vibration in mm/s, single digits11service = (vib > 4.5).astype(float)                  # only vibration decides it12raw = np.column_stack([rpm, vib])13scaled = (raw - raw[:600].mean(axis=0)) / raw[:600].std(axis=0)   # stats from training rows only14for name, data in (("raw", raw), ("scaled", scaled)):15    pred = vote(data[:600], service[:600], data[600:], 5)16    print(f"{name:>6}: accuracy {(pred == service[600:]).mean():.3f}")17# ->    raw: accuracy 0.59018# -> scaled: accuracy 0.985

The raw model lands barely above a coin flip because it chose neighbours by motor speed. One line of arithmetic takes it to 0.985. No hyperparameter search recovers that gap.

Watch Out For

Fitting the scaler before splitting the data

The tempting order is to standardise the full table and then split, because it's one line instead of two. The mean and standard deviation applied to your test rows now include those rows, so held-out information walked into every neighbour decision.

The symptom is a validation score you can't reproduce in production, and it worsens as the dataset shrinks. Fit on training rows only, as the snippet does with raw[:600], and refit per fold inside cross-validation. Distance-based models feel this leakage hardest, because scaling isn't a nicety here — it's load-bearing.

Running it on a wide feature space

You have 400 sensor channels per pump, so you feed in all 400. Accuracy is mediocre. You sweep k from 1 to 50 and the curve is almost flat.

That flatness is the tell, and it means something specific: the neighbour set is noise at every k, so averaging more of it changes nothing. Fix the geometry, not the vote. Cut to the channels that separate the classes, or reduce dimensions with PCA and run neighbours on the components. If neither helps, switch models — a tree ensemble ignores irrelevant columns for free.

The Quick Version

Store the training set, then classify a query by the majority vote of its k closest rows.
The distance function is the model. Euclidean assumes comparable axes, Manhattan resists one large coordinate, Minkowski dials between them, cosine drops magnitude.
Scale every feature first. One unscaled wide-range column decides every neighbour on its own.
Small k follows noise; large k smooths toward the majority. Keep it odd for two classes.
Prediction is linear in training-set size per query, and the whole set lives in memory.
In high dimensions nearest and farthest converge, so neighbours stop meaning anything.