Face Detection and Recognition

Face recognition runs as a pipeline: a detector boxes each face, alignment standardizes the crop, and an embedding network names it by nearest neighbour.

The detector boxes the face, alignment standardizes the crop, and the embedding scores 0.9973 against Alice but -0.266 against Bob, so the 0.85 threshold names Alice

Why Does This Exist?

Finding faces and naming them are two different problems stapled together. Detection scans a messy scene with changing light, angles, and occlusions and answers "where are the faces," exactly the localization job object detection describes. Recognition takes one cropped face and answers "whose is it" among people the system may never have trained on. One network cannot do both well: the first needs translation-heavy scanning, the second needs identity-preserving compression. The pipeline splits them so each stage trains on its own objective.

Think of It Like This

The bouncer and the ID checker

A busy venue posts a bouncer at the door and an ID checker behind a desk. The bouncer scans the crowd, pulls out each person, and frames them in the doorway; names never cross the bouncer's mind. The checker studies one framed face at a time and matches it against the guest list.

The bouncer is the detector, the doorway framing is alignment, and the guest list is the embedding database. Where it stops: a checker reads names off cards, while the network compresses the face into numbers first, so a bad crop corrupts the numbers and no guest list fixes it.

How It Actually Works

1. Detection draws the boxes

A face detector slides over the image and outputs one bounding box per face plus a confidence, exactly like a generic detector with a single class. Modern variants also emit landmarks through face landmark detection: eye centers, nose tip, mouth corners. Overlapping proposals collapse to one box each by suppression, and the survivors define the crops everything downstream sees.

2. Alignment standardizes the crop

The landmarks warp each crop so eyes and mouth land in fixed positions, removing most pose variation before recognition starts. Skipping this step forces the embedding network to learn pose invariance from data instead, which costs far more training faces for worse accuracy.

3. Embeddings name by nearest neighbour

The aligned crop passes through a network trained by triplet loss, siamese networks, or margin losses like ArcFace, which pull same-person vectors together and push different people apart. Identity is then geometry: similarity $s = (a \cdot b) / (\lVert a \rVert \lVert b \rVert)$ , and the face takes the database name with the highest score above a threshold $\tau$ .

Worked example

The detector boxes a face and alignment standardizes it. The network emits the 4-number toy embedding $u = [0.78, 0.15, 0.48, -0.15]$ against Alice $a = [0.8, 0.1, 0.5, -0.2]$ and Bob $b = [-0.5, 0.9, 0.1, 0.8]$ . Against Alice: dot $0.624 + 0.015 + 0.24 + 0.03 = 0.909$ , norms $0.9401$ and $0.9695$ , so $s = 0.909 / (0.9401 \times 0.9695) = 0.9973$ . Against Bob: dot $-0.39 + 0.135 + 0.048 - 0.12 = -0.327$ , norm $1.3077$ , so $s = -0.327 / (0.9401 \times 1.3077) = -0.266$ . With $\tau = 0.85$ , Alice clears it and Bob never comes close: recognized Alice. One-to-one confirmation against a claimed identity instead is the separate job in face verification.

Code

Stdlib Python replays the worked match: two cosine similarities and the threshold decision.

1import math2
3alice = [0.8, 0.1, 0.5, -0.2]4bob = [-0.5, 0.9, 0.1, 0.8]5unknown = [0.78, 0.15, 0.48, -0.15]6
7def cos(a, b):8    dot = sum(p * q for p, q in zip(a, b))9    return dot / (math.sqrt(sum(p * p for p in a)) * math.sqrt(sum(q * q for q in b)))10
11sa = cos(unknown, alice)12sb = cos(unknown, bob)13print(round(sa, 4), round(sb, 4))14# -> 0.9973 -0.26615print("Alice" if sa > 0.85 else "unknown")16# -> Alice

Real embeddings run 128 to 512 dimensions, but the arithmetic is identical: normalize, dot, threshold. Enrolling a newcomer means storing one vector, never retraining.

Watch Out For

Training a classifier instead of an embedding

A softmax over the people in the training set cannot name anyone new without retraining the whole model. The symptom is a rebuild every time a face enrolls. Fix: train metric embeddings with triplet or margin losses, so enrollment writes one vector to the database and recognition stays nearest neighbour.

Skipping alignment and blaming the embedding

Raw tilted crops inflate intra-person distances, so the same person scores like two strangers. The symptom is accuracy that collapses on angled faces while frontal pairs look fine. Fix: align every crop with detected landmarks before embedding, and only then tune the threshold.

The Quick Version

Detection boxes each face and alignment standardizes the crop via landmarks.
The embedding network compresses identity into a vector trained by triplet or margin losses.
Identity is nearest neighbour under cosine similarity with a threshold $\tau = 0.85$ .
The toy match scores $0.9973$ for Alice and $-0.266$ for Bob, so Alice wins.
Enrollment stores one vector; classifiers that need retraining per person are the wrong tool.