Support Vector Machines
Maximum margin — finding the widest street between two classes.
Many lines can perfectly separate two classes. However, lines that pass too close to the training data are brittle and likely to misclassify new examples.
The Maximum-Margin Hyperplane
An SVM finds the "widest street" that separates the groups. It positions the boundary exactly in the middle of this maximum margin for maximum safety.
Support Vectors
The points sitting directly on the edge of the street are the support vectors. They alone define the boundary; all other data points could be deleted without changing the model.
Soft Margins
When classes overlap, a rigid boundary fails. Slack variables allow a few points to violate the margin or even end up on the wrong side, preserving a wider, more generalisable street.
Where It Breaks
For non-linear data, SVMs use the kernel trick. But choosing the right kernel and tuning its hyperparameters (like C and gamma) is difficult and opaque.
The Quick Version
- Problem: Arbitrary boundaries are brittle on unseen data.
- Mechanism: Find the widest possible margin between classes.
- Support Vectors: Only the edge points define the model.
- Soft Margins: Allow some violations to avoid overfitting.
- Failure: Kernels can be black boxes that overfit easily.