Dot Product

Multiply two lists of numbers position by position and add up the result — a single number saying how much the two point the same way.

Why Does This Exist?

Every time a search over embeddings returns a result, a dot product decided it. Every attention score inside a transformer is a dot product. Every neuron in a dense layer computes one before it does anything else. It is, in raw arithmetic performed on GPUs, very likely the single most-executed operation in machine learning.

The reason it earns that position is that it answers a question no individual number can: do these two things point the same way? Two embeddings of 768 numbers each are impossible to compare by eye. One dot product collapses them to a single score, and that score behaves the way intuition wants — large when the vectors agree, zero when they are unrelated, negative when they oppose.

And it comes with a trap attached, which is the practical reason to read past the definition. Because the dot product mixes direction with magnitude, using it as a similarity score without controlling magnitude produces a retrieval system that quietly ranks by vector length. That bug is common, invisible in unit tests, and fixed by one line.

Think of It Like This

Pushing a trolley at an angle

You are pushing a supermarket trolley. The trolley only rolls forward, but your push comes in at an angle.

Push straight ahead and all of your effort goes into moving it. Push at forty-five degrees and some of your effort is wasted sideways into the handle. Push directly sideways and the trolley does not move at all — you are working hard and achieving nothing forward.

The dot product is exactly the useful part of your push: how much of your effort lies along the direction that matters. Push harder and it grows. Turn more sideways and it shrinks. Turn a full ninety degrees and it hits zero, which is what orthogonal means. Turn further and it goes negative, because you are now pulling the trolley backwards.

The analogy stops at dimensions: a trolley splits effort into two directions, while an embedding dot product splits agreement across hundreds of coordinates at once.

How It Actually Works

There are two formulas, they always agree, and each is useful for a different reason.

The computational one — multiply matching positions, add everything up:

a \cdot b = \sum_{i=1}^{n} a_i b_i

In plain words: pair up the components, multiply each pair, total the results. Both vectors must have the same length $n$ , and the answer is always a single scalar — shapes $(n,)$ and $(n,)$ give shape $()$ .

The geometric one — length times length times the angle between them:

a \cdot b = \lVert a \rVert \, \lVert b \rVert \cos\theta

In plain words: the sizes of the two vectors, scaled down by how much they disagree in direction. The $\lVert \cdot \rVert$ bars are the L2 length from norms, and $\theta$ is the angle between the vectors.

The second formula is where all the meaning lives. Because $\cos\theta$ runs from $1$ down to $-1$ :

Same direction ( $\theta = 0$ , $\cos\theta = 1$ ) — the maximum possible value.
At right angles ( $\theta = 90°$ , $\cos\theta = 0$ ) — exactly zero. The vectors are orthogonal and share nothing.
Opposite ( $\theta = 180°$ , $\cos\theta = -1$ ) — maximally negative.

The magnitude problem, and cosine similarity

Look hard at that second formula, because it contains the trap. The dot product is $\lVert a \rVert \lVert b \rVert \cos\theta$ — two length terms and one direction term. A long vector pointing roughly the wrong way can easily beat a short vector pointing exactly the right way.

If what you want is direction only, divide the lengths out:

\cos\theta = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}

That is cosine similarity, and it lands in $[-1, 1]$ regardless of how long either vector is. It is the right default for comparing embeddings, because embedding magnitude is usually an artifact of training rather than a meaning you want to rank by.

There is a shortcut production systems rely on. If you normalise every vector to length $1$ first — making each a unit vector — then $\lVert a \rVert \lVert b \rVert = 1$ and the plain dot product is the cosine. This is why vector databases store normalised embeddings: cosine semantics at dot-product speed.

Worked example

Take $a = [4, 3]$ and $b = [5, 0]$ .

Hand-computed componentwise: $a \cdot b = (4 \times 5) + (3 \times 0) = 20 + 0 = 20$ .

Now check it geometrically. Lengths are $\lVert a \rVert = \sqrt{16 + 9} = \sqrt{25} = 5$ and $\lVert b \rVert = \sqrt{25 + 0} = 5$ . Since $b$ lies flat along the horizontal axis, the angle of $a$ above it has $\cos\theta = 4/5 = 0.8$ . So $\lVert a \rVert \lVert b \rVert \cos\theta = 5 \times 5 \times 0.8 = 25 \times 0.8 = 20$ . The two formulas agree, as they must.

The projection is the piece the diagram draws: how far along $b$ direction $a$ reaches, which is $\lVert a \rVert \cos\theta = 5 \times 0.8 = 4$ . Equivalently $a \cdot b / \lVert b \rVert = 20/5 = 4$ . That is $a$ shadow on $b$ .

And the cosine similarity is $20 / (5 \times 5) = 20/25 = 0.8$ — a strong match, though not a perfect one.

Code

The final line is the whole lesson: scaling changes the dot product and leaves cosine untouched.

1import numpy as np2
3
4def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:5    # Divide out both magnitudes so only the angle survives.6    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))7
8
9a = np.array([4.0, 3.0])  # shape (2,)10b = np.array([5.0, 0.0])  # shape (2,)11
12print(a @ b)  # -> 20.0   the dot product13print(np.linalg.norm(a), np.linalg.norm(b))  # -> 5.0 5.014print(cosine_similarity(a, b))  # -> 0.815print(a @ b / np.linalg.norm(b))  # -> 4.0   projection of a onto b16
17# Length hijacks the raw dot product: 10*b agrees no better, but scores 10x.18print(a @ (10 * b), cosine_similarity(a, 10 * b))  # -> 200.0 0.8

Scaling $b$ by ten multiplies the dot product by ten to $200$ while cosine stays $0.8$ , because the angle never changed.

Watch Out For

Ranking by dot product on unnormalised embeddings

Embedding magnitudes are not uniform. Longer documents, more frequent tokens, and certain fine-tuning regimes all produce systematically longer vectors. Rank by raw dot product and those long vectors win more often — not because they are more relevant, but because they are longer.

What makes it hard to catch: the results are not garbage. They are plausible and mostly reasonable, just worse than they should be. No test fails. The usual symptom is a handful of documents that keep surfacing for unrelated queries.

The fix is one line at write time: normalise every vector to length $1$ before storage, then the dot product is cosine similarity and you keep the speed. Do it at ingestion and confirm it — np.linalg.norm(v) should be $1.0$ for everything in the index.

Accumulating a long dot product in low precision

A dot product over 4096 dimensions is 4096 multiplications summed into one accumulator, and in reduced precision that accumulator is the weak point.

float16 maxes out around 65504. Attention scores over a large hidden dimension can exceed that before any softmax normalises them, and the sum becomes inf. From there softmax produces NaN, and one NaN contaminates every parameter it reaches backward. This is a genuine failure mode of naive mixed-precision attention.

Even short of overflow, adding thousands of small float16 numbers loses low-order bits at every step. This is why attention divides scores by $\sqrt{d_k}$ before softmax, and why kernels accumulate in float32 even when inputs are float16. If you hand-roll a kernel, accumulate wider than you store.

The Quick Version

Multiply matching components and sum: two vectors of length $n$ give one scalar.
Equivalently $\lVert a \rVert \lVert b \rVert \cos\theta$ — two magnitude terms and one direction term.
Positive means broadly aligned, zero means orthogonal, negative means opposed.
Because magnitude is baked in, raw dot products rank partly by vector length.
Divide by both lengths to get cosine similarity, which measures direction alone and lands in $[-1, 1]$ .
Normalise vectors once at storage time and the dot product becomes cosine for free. This is what vector databases do.
The projection of $a$ onto $b$ is $a \cdot b / \lVert b \rVert$ , here $20/5 = 4$ .
Accumulate long dot products in wider precision than you store them in.