Skip to content
AI360Xpert
Visual Explainers
Visual explainer

K-Nearest Neighbors

Predict a new point's class by looking at the k training points closest to it and taking a majority vote.

We have a new point and need to decide which group it belongs to.
We have a new point and need to decide which group it belongs to.

Unlike most machine learning algorithms, k-NN doesn't actually build a model or learn any parameters during training. It just memorises the entire dataset and waits for a new arrival.

Copy the Closest

The simplest answer is to find the single closest point and copy its class.
The simplest answer is to find the single closest point and copy its class.

The simplest version is to just find the one single training point that sits closest to the new arrival, and assign its class. This is 1-NN. It's fast, but highly vulnerable to a single noisy point in the wrong place.

Take a Vote

To avoid being fooled by a single noisy point, we ask the 'k' closest neighbors to vote.
To avoid being fooled by a single noisy point, we ask the 'k' closest neighbors to vote.

To get a more stable prediction, you look at the kk closest neighbors instead. If k=3k=3, the algorithm finds the three closest points and takes a majority vote. The new point becomes whatever class holds the majority in its immediate neighborhood.

Free Complexity

Because the model just looks up answers locally, the overall boundary can wrap perfectly around any shape.
Because the model just looks up answers locally, the overall boundary can wrap perfectly around any shape.

Because there is no global mathematical formula trying to draw a line, k-NN can easily draw highly complex, non-linear decision boundaries. It naturally wraps around clusters of any shape, creating islands and peninsulas exactly where the data demands them.

Lost in High Dimensions

In high dimensions, every point is far away from every other point, making 'nearest' meaningless.
In high dimensions, every point is far away from every other point, making 'nearest' meaningless.

When you have hundreds or thousands of features, the math of distance breaks down. This is the curse of dimensionality: in high-dimensional space, everything is pushed out to the corners, so the "nearest" neighbor is just as far away as a randomly chosen one, and the vote becomes a guess.

The Quick Version

  • Training just means memorising the data.
  • It finds the k closest points to the new input.
  • The neighbors take a majority vote.
  • It draws perfectly complex boundaries for free.
  • It fails completely on high-dimensional data.