Orthogonality and Projections

Perpendicular means zero dot product. Projection splits any vector into a parallel part plus a perpendicular leftover, which is how clean bases get built.

Vector v equals (4, 3) splits into projection (4, 0) along u plus residual (0, 3) perpendicular to u, so u dot r equals 0

Why Does This Exist?

Least squares, Fourier series, PCA, and attention all throw away the part they cannot use and keep the part they can. Without a clean split, the leftover leaks into the kept part: correlated features double count evidence, non-orthogonal bases entangle coefficients, and denoising removes signal with the noise.

Orthogonality makes the split clean. Two vectors are orthogonal when their dot product is zero, meaning they share no direction. Projection drops one vector onto another (or onto a subspace), leaving a residual that is orthogonal to everything kept. Lengths come from norms, and the full basis-building payoff is PCA derivation. This page covers the split, the formula, and the 3-4-5 check.

The one idea you need first

For unit direction $\hat{u}$ with $\hat{u} \cdot \hat{u} = 1$ , the projection of $v$ along $\hat{u}$ is $(v \cdot \hat{u})\hat{u}$ : how far $v$ reaches in that direction, laid back along it. The residual $r = v - \text{proj}$ is perpendicular by construction. For general $u$ , divide by the squared length: $\text{proj}_u(v) = ((v \cdot u)/(u \cdot u))u$ .

Think of It Like This

Noon shadow on a straight road

Sun directly overhead throws your shadow straight onto the road centerline. Your full height is the vector $v$ ; the shadow on the road is the projection; the vertical drop from your head to the shadow tip is the residual, perpendicular to the road.

The shadow length tells how far along the road you reach, nothing about height. Where it stops: shadows collapse a dimension, so many heights share one shadow. Projection likewise keeps one part and parks the rest in the residual, recoverable only by adding it back.

How It Actually Works

Orthogonal means $u \cdot r = 0$ . Projection keeps the parallel part and parks the rest: $v = \text{proj}_u(v) + r$ with $u \cdot r = 0$ . Only the parallel part survives a second projection, which is why orthogonal bases decouple coefficients. Gram-Schmidt repeats this: take the next vector, subtract its projections onto every kept direction, keep the residual. Each residual is orthogonal to all previous ones by construction.

Worked example

Take $v = (4, 3)$ and unit direction $\hat{u} = (1, 0)$ along the x-axis. Dot: $v \cdot \hat{u} = 4 \times 1 + 3 \times 0 = 4$ . Projection: $\text{proj} = 4\hat{u} = (4, 0)$ . Residual: $r = (4, 3) - (4, 0) = (0, 3)$ . Check: $\hat{u} \cdot r = 1 \times 0 + 0 \times 3 = 0$ , so perpendicular. Lengths: $\lVert v \rVert = \sqrt{16 + 9} = 5$ , $\lVert \text{proj} \rVert = 4$ , $\lVert r \rVert = 3$ , and $4^2 + 3^2 = 25 = 5^2$ . The diagram draws exactly this right triangle.

Code

1import numpy as np2
3
4def project(v: np.ndarray, u: np.ndarray) -> np.ndarray:5    return ((v @ u) / (u @ u)) * u  # scale by overlap over squared length6
7
8v = np.array([4.0, 3.0])9u = np.array([1.0, 0.0])10p = project(v, u)11print(p.tolist())  # -> [4.0, 0.0]12print((v - p).tolist())  # -> [0.0, 3.0]13print(float(u @ (v - p)))  # -> 0.0

Parallel part plus perpendicular leftover, with zero dot product confirming the split.

Watch Out For

Using the unit formula on a non-unit direction

$(v \cdot u)u$ without dividing by $u \cdot u$ scales the answer by $\lVert u \rVert^2$ . With $u = (2, 0)$ , it reports $(8, 0)$ instead of $(4, 0)$ , and the residual $(-4, 3)$ is not perpendicular: $u \cdot r = -8$ .

Symptom: residual dot product nonzero, lengths violating Pythagoras. Fix: always use $((v \cdot u)/(u \cdot u))u$ , or normalize first and use $(v \cdot \hat{u})\hat{u}$ . Test on $(4, 3)$ against $(1, 0)$ where the answer must be $(4, 0)$ .

Projecting onto correlated axes one at a time

Subtracting the projection onto $u_1$ then onto $u_2$ works only when $u_1 \cdot u_2 = 0$ . When axes correlate, the second subtraction reintroduces a $u_1$ component, so coefficients entangle and the residual still points along kept directions.

Symptom: residual dot products against kept axes are nonzero, and refitting changes earlier coefficients. Fix: orthogonalize first (Gram-Schmidt), or solve the joint normal equations instead of projecting sequentially.

The Quick Version

Orthogonal means zero dot product: no shared direction.
Projection splits $v$ into parallel $(v \cdot \hat{u})\hat{u}$ plus residual perpendicular to $u$ .
Example $(4, 3)$ along $(1, 0)$ gives $(4, 0)$ plus $(0, 3)$ with dot zero and lengths 3 4 5.
General formula divides by squared length: $((v \cdot u)/(u \cdot u))u$ .
Gram-Schmidt repeats the split to build orthogonal bases; PCA lives on the same split.
Unit formula on non-unit vectors and sequential projection onto correlated axes both silently fail.