Support Vector Machines

Of every line that splits two classes, keep the one farthest from the nearest point on each side. A few border points decide everything, and there is no probability in the answer.

Of every line that splits the two classes, the widest-margin line sits farthest from the closest point on each side, and only the three ringed border points decide where it goes

Why Does This Exist?

When two groups separate cleanly, thousands of boundaries get every training point right. Logistic regression draws one and stops once the error hits zero. The boundary you receive can pass a hair from a real point. Correct today, fragile tomorrow. A worn sensor or a shifted scale pushes new points across.

One term first. A hyperplane is the flat dividing surface a linear model draws: a line on a plane, a plane in 3D, the same idea higher up. Most linear models draw one and never ask where it sits. This one does. Of the many separating hyperplanes, take the one farthest from the nearest point on either side. The problem is convex, so there is exactly one answer and two runs agree.

This page covers that widest-street boundary, its margin math, and its loss. Nonlinear separation through feature maps belongs to the kernel trick.

Think of It Like This

Mowing the widest path between two hedges

Two tall hedges face each other and you must mow one straight path between them, the widest that fits, so a walker can pass without getting scratched. Nudge the center line left and the path narrows. Nudge it right and it narrows again. Exactly one line leaves the most room, running down the middle of the tightest gap.

Now notice which branches placed it: two or three tips poking out furthest. Trim a branch deep inside either hedge and nothing moves. Trim one brushing the path edge and the whole path shifts.

Where the picture stops: hedges sit still, data does not. If one poking branch was a fluke, your perfect path is wrong, which is why the soft version below lets a few branches cross the edge for a fee.

How It Actually Works

The margin and its width

Score a point as $f(x) = w^T x + b$ , where $x$ holds the features, $w$ holds one weight per feature, and $b$ shifts the surface. The boundary is the set where $f(x) = 0$ . Label the classes $-1$ and $+1$ and demand the right sign with room to spare:

y_i (w^T x_i + b) \geq 1 \quad \text{for every point } i

The surfaces where the score equals $-1$ and $+1$ are the margin boundaries, dashed in the diagram. The gap between them measures $2 / \|w\|$ , with $\|w\|$ the length of the weight vector, so widening the street means shortening $w$ . Work it: $w = (1, 0)$ has length $1$ and margin $2.0 / 1 = 2.0$ , while $w = (2, 0)$ has length $2$ and margin $2.0 / 2 = 1.0$ . Shorter weights, wider street.

This strict form is the hard margin, and it insists on perfect separation. One damaged point inside enemy territory makes every constraint unsatisfiable at once, and the optimizer returns nothing instead of something worse.

Soft margin and the C dial

The repair lets points break the rule and charges per break. Each point gets a slack $\xi_i$ measuring its shortfall, and the objective becomes $\frac{1}{2}\|w\|^2 + C \sum_i \xi_i$ . Read $C$ as the price of a violation, because everyone gets its direction backwards at first.

Large $C$ makes violations expensive, so the boundary contorts to satisfy nearly every point and the street narrows. Small $C$ makes violations cheap, so the optimizer buys a wide street and misclassifies some training points on purpose. Turning $C$ up regularizes less, the opposite of most dials.

Hinge loss and worked numbers

The per-point charge has a name. For score $f(x)$ and label $y$ :

L = \max(0, 1 - y f(x))

Read it as a rule about when the model stops caring. Score a point right and clear of the margin and the loss is exactly zero, not small. Inside the margin or on the wrong side, the cost climbs in a straight line. Three cases: $y = 1, f = 0.5$ gives $\max(0, 0.5) = 0.5$ ; $y = 1, f = 2.0$ gives $\max(0, -1.0) = 0.0$ ; $y = -1, f = 0.5$ gives $\max(0, 1.5) = 1.5$ .

That cutoff is the defining property. Points safely beyond the margin contribute nothing, so the answer rests on the handful inside or touching it: the support vectors, three in the diagram. Delete every other point and refit for an identical boundary. Compact, and only as trustworthy as those few rows. Note the output throughout is a signed distance to a surface, never a probability.

Code

Margin widths and hinge charges for the worked numbers, so you can check the direction of $C$ reasoning yourself.

1import math2
3
4def margin_width(w):5    return 2.0 / math.sqrt(w[0] ** 2 + w[1] ** 2)6
7
8print(margin_width((1.0, 0.0)))  # -> 2.09print(margin_width((2.0, 0.0)))  # -> 1.010
11
12def hinge(y, f):13    return max(0.0, 1 - y * f)14
15
16print(hinge(1, 0.5))  # -> 0.517print(hinge(1, 2.0))  # -> 0.018print(hinge(-1, 0.5))  # -> 1.5

Longer weights mean a narrower street, and only points short of the margin pay anything.

Watch Out For

Turning C the wrong way when the model overfits

Symptom: the boundary hugs the data, validation slips, and someone reasons that $C$ is a regularization strength, raises it from 1 to 100, and watches validation slip further.

Fix: memorize the price reading instead. Fit at two $C$ values and compare margin width or support-vector count. Narrower street and fewer support vectors mean less regularization, every time. Log both numbers in the sweep so direction never relies on memory.

Reading the decision score as a probability

Symptom: scores get pushed through a squashing function, the rankings look right, accuracy looks healthy, and any decision that multiplies the number by a cost runs on fiction. Change $C$ and every pseudo probability moves, because the input was a distance in units of $\|w\|$ .

Fix: fit the mapping on held-out data instead of inventing it. Keep the raw distance for ranking and learn the probability map from a separate split before any cost math touches it.

The Quick Version

Of all separating hyperplanes, this keeps the one farthest from the closest point on each side, and convexity gives a single answer.
The street width is $2 / \|w\|$ , so maximizing it means minimizing the weight vector length.
Hard margin needs perfect separation. One outlier makes it infeasible, so the soft form buys violations at price $C$ .
Large $C$ means pricey violations and a narrow street. Small $C$ means a wide street that tolerates errors.
Hinge loss is exactly zero past the margin, so only nearby points shape the boundary. Those are the support vectors.
The output is a signed distance, not a probability. Calibrate before multiplying by costs.