Dot Product

Why Does This Exist?

Every time a search over embeddings returns a result, a dot product decided it. Every attention score inside a transformer is a dot product. Every neuron in a dense layer computes one before it does anything else. It is, in terms of raw arithmetic performed on the planet's GPUs, very likely the single most-executed operation in machine learning.

The reason it earns that position is that it answers a question no individual number can: do these two things point the same way? Two embeddings of 768 numbers each are impossible to compare by eye. One dot product collapses them to a single score, and that score behaves the way intuition wants — large when the vectors agree, zero when they are unrelated, negative when they oppose.

And it comes with a trap attached, which is the practical reason to read past the definition. Because the dot product mixes direction with magnitude, using it as a similarity score without controlling magnitude produces a retrieval system that quietly ranks by vector length. That bug is common, it is invisible in unit tests, and it is fixed by one line.

Think of It Like This

How It Actually Works

There are two formulas, they always agree, and each is useful for a different reason.

The computational one — multiply matching positions, add everything up:

In plain words: pair up the components, multiply each pair, total the results. Both vectors must have the same length $n$, and the answer is always a single scalar — shapes $(n,)$ and $(n,)$ give shape $()$.

The geometric one — length times length times the angle between them:

In plain words: the sizes of the two vectors, scaled down by how much they disagree in direction. The $\lVert \cdot \rVert$ bars are the L2 length from norms, and $\theta$ is the angle between the vectors.

The second formula is where all the meaning lives. Because $\cos\theta$ runs from 1 down to $-1$:

• Same direction ($\theta = 0$, $\cos\theta = 1$) — the maximum possible value.
• At right angles ($\theta = 90°$, $\cos\theta = 0$) — exactly zero. The vectors are orthogonal and share nothing.
• Opposite ($\theta = 180°$, $\cos\theta = -1$) — maximally negative.

The magnitude problem, and cosine similarity

Look hard at that second formula, because it contains the trap. The dot product is $\lVert a \rVert \lVert b \rVert \cos\theta$ — two length terms and one direction term. A long vector pointing roughly the wrong way can easily beat a short vector pointing exactly the right way.

If what you want is direction only, divide the lengths out:

That is cosine similarity, and it lands in $[-1, 1]$ regardless of how long either vector is. It is the right default for comparing embeddings, because embedding magnitude is usually an artifact of training rather than a meaning you want to rank by.

There is a shortcut worth knowing, because production systems rely on it. If you normalise every vector to length 1 first — making each a unit vector — then $\lVert a \rVert \lVert b \rVert = 1$ and the plain dot product is the cosine. This is why vector databases store normalised embeddings: it buys cosine semantics at dot-product speed.

Worked example

Take $a = [4, 3]$ and $b = [5, 0]$.

Componentwise: $a \cdot b = (4 \times 5) + (3 \times 0) = 20$.

Now check it geometrically. The lengths are $\lVert a \rVert = \sqrt{16 + 9} = 5$ and $\lVert b \rVert = 5$. Since $b$ lies flat along the horizontal axis, the angle of $a$ above it has $\cos\theta = 4/5 = 0.8$. So $\lVert a \rVert \lVert b \rVert \cos\theta = 5 \times 5 \times 0.8 = 20$. The two formulas agree, as they must.

The projection is the piece the diagram draws: how far along $b$'s direction $a$ reaches, which is $\lVert a \rVert \cos\theta = 5 \times 0.8 = 4$. Equivalently $a \cdot b / \lVert b \rVert = 20/5 = 4$. That is $a$'s shadow on $b$.

And the cosine similarity is $20 / (5 \times 5) = 0.8$ — a strong match, though not a perfect one.

Show Me the Code

1import numpy as np2
3
4def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:5    # Divide out both magnitudes so only the angle survives.6    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))7
8
9a = np.array([4.0, 3.0])   # shape (2,)10b = np.array([5.0, 0.0])   # shape (2,)11
12print(a @ b)                              # -> 20.0   the dot product13print(np.linalg.norm(a), np.linalg.norm(b))  # -> 5.0 5.014print(cosine_similarity(a, b))            # -> 0.815print(a @ b / np.linalg.norm(b))          # -> 4.0   projection of a onto b16
17# Length hijacks the raw dot product: 10*b agrees with a no better, but scores 10x.18print(a @ (10 * b), cosine_similarity(a, 10 * b))  # -> 200.0 0.8

That final line is the whole lesson. Scaling b by ten multiplies the dot product by ten while the cosine similarity does not budge, because the two vectors still point at exactly the same angle to each other.

Watch Out For

The Quick Version

• Multiply matching components and sum: two vectors of length $n$ give one scalar.
• Equivalently $\lVert a \rVert \lVert b \rVert \cos\theta$ — two magnitude terms and one direction term.
• Positive means broadly aligned, zero means orthogonal, negative means opposed.
• Because magnitude is baked in, raw dot products rank partly by vector length.
• Divide by both lengths to get cosine similarity, which measures direction alone and lands in $[-1, 1]$.
• Normalise vectors once at storage time and the dot product becomes cosine for free. This is what vector databases do.
• The projection of $a$ onto $b$ is $a \cdot b / \lVert b \rVert$.
• Accumulate long dot products in wider precision than you store them in.

What to Read Next

• Norms define the lengths this page divides by, and explain why L2 is the one used here.
• Matrix Multiplication is this operation done many times at once — every entry of the result is one dot product.
• Vectors, Matrices and Tensors covers the shape rules that decide whether two things can even be dotted.
• Definitions worth a look: Cosine Similarity, Unit Vector, Orthogonal Vectors, and Euclidean Distance.
• Related interview question: Why normalise vectors before computing cosine similarity?