Inner Product Spaces

A space with a dot product: the four rules that buy angles, lengths and projections in any dimension, including spaces of functions.

The inner product splits into length times length times cosine, so the 11 from 1 2 dot 3 4 must sit below 2.236 times 5 and it does

Why Does This Exist?

Vector spaces give addition and scaling but no geometry: without extra structure you cannot say how long a vector is, whether two vectors are perpendicular, or how much of one points along another. An inner product adds exactly that: a pairing $\langle u, v \rangle$ obeying four axioms, from which lengths, angles, projections and the Cauchy-Schwarz inequality all follow.

This abstraction pays off the moment vectors stop being arrows. Function spaces (signals, images, solutions of differential equations) carry inner products defined by integrals. Kernel methods evaluate inner products in feature spaces nobody constructs explicitly. The dot product is the prototype; this page is the general contract it satisfies, which is what lets the same geometry run on functions.

The one idea you need first

A vector space supports adding vectors and scaling them, covered in vector spaces and rank. Lengths come from norms. The inner product sits underneath both: it manufactures a norm via $\lVert v \rVert = \sqrt{\langle v, v \rangle}$ , and that norm manufactures a distance.

Think of It Like This

A tape measure that also reads angles

A ruler measures lengths along one line. A protractor measures the opening between two lines. An inner product is the single instrument that does both and agrees with each: feed it one vector twice for squared length, feed it two different vectors for length-times-length-times-cosine.

Carry that instrument into a function space and it keeps working: "length" of a signal becomes root-mean-square energy, "angle" between two signals becomes their correlation. Same four axioms, wildly different objects measured. The contract travels; the pictures stay behind.

How It Actually Works

The four axioms for a real inner product:

Symmetry: $\langle u, v \rangle = \langle v, u \rangle$ (conjugate symmetry over complex numbers).
Linearity: linear in each argument: $\langle au + bw, v \rangle = a\langle u, v \rangle + b\langle w, v \rangle$ .
Positive definiteness: $\langle v, v \rangle \ge 0$ , with equality only for $v = 0$ .

From these alone: the induced norm satisfies the triangle inequality, projections $P_v(u) = \langle u, v \rangle v / \langle v, v \rangle$ behave, and angles make sense via $\cos\theta = \langle u, v \rangle / (\lVert u \rVert \lVert v \rVert)$ . The crown jewel is Cauchy-Schwarz:

|\langle u, v \rangle| \le \lVert u \rVert \lVert v \rVert

with equality exactly when $u$ and $v$ are linearly dependent. Every "correlation is at most 1" statement is this inequality in disguise.

Worked example

Test Cauchy-Schwarz on $u = [1, 2]$ and $v = [3, 4]$ with the standard dot product:

\langle u, v \rangle = 1 \cdot 3 + 2 \cdot 4 = 11, \quad \lVert u \rVert = \sqrt{5} \approx 2.236, \quad \lVert v \rVert = 5

Check: $11 \le 2.236 \times 5 = 11.18$ . Holds with a small margin, and the margin says the vectors are nearly (not exactly) aligned: $\cos\theta = 11/11.18 \approx 0.984$ . The same computation with integrals, $\int f g \le \sqrt{\int f^2 \int g^2}$ , bounds signal correlations identically.

Code

The point is the check: verify Cauchy-Schwarz and the projection formula on concrete vectors.

1import numpy as np2
3
4u = np.array([1.0, 2.0])5v = np.array([3.0, 4.0])6print(float(u @ v) <= float(np.linalg.norm(u) * np.linalg.norm(v)))  # -> True7proj = (u @ v) / (v @ v) * v  # projection of u onto v8print(proj.tolist())  # -> [1.32, 1.76]

The projection $[1.32, 1.76]$ points along $v$ (it equals $0.44 \times [3, 4]$ ), and the leftover $u -$ proj is perpendicular to $v$ : their dot product is $0$ up to rounding, which allclose confirms.

Watch Out For

Assuming the dot product is the only inner product

Weighted inner products $\langle u, v \rangle_W = u^T W v$ , the Frobenius pairing $\mathrm{tr}(A^T B)$ , and integral pairings all satisfy the axioms and induce different geometries: different lengths, different angles, different "closest points". Least squares with a weight matrix is exactly a change of inner product. Name which pairing a result uses before quoting its geometry.

Forgetting conjugate symmetry over complex numbers

Over $\mathbb{C}$ the axioms require $\langle u, v \rangle = \overline{\langle v, u \rangle}$ , and physics-style texts put linearity in the second argument instead of the first. Code mixing conventions (NumPy's vdot conjugates the first argument) silently flips phases. In complex spaces, check the convention of every routine you call.

The Quick Version

Inner products add lengths, angles and projections to vector spaces through four axioms.
Cauchy-Schwarz bounds every pairing by the product of lengths: 11 below 11.18 above.
The induced norm manufactures distances, so geometry follows the axioms everywhere.
Function spaces carry integral inner products; kernels evaluate them implicitly.
Weighted, Frobenius and integral pairings are all legitimate: name yours.