Orthogonal Vectors
Two vectors at right angles, whose dot product is exactly zero.
Think of It Like This
Pushing a trolley directly sideways: maximum effort, zero forward progress.
Orthogonal means . Since the dot product equals and the magnitudes are nonzero, a zero product forces , so the angle is 90 degrees. Neither vector has any component along the other.
In high dimensions this becomes surprisingly common: two random vectors in 768 dimensions are nearly orthogonal with high probability, which is what lets an embedding space store many concepts that barely interfere with each other. It is also why a cosine similarity near zero is the expected result for unrelated items rather than a suspicious one.
A set of mutually orthogonal unit vectors is called orthonormal, and those make ideal bases — projecting onto them is just a dot product, with no correction needed. Orthogonal initialisation schemes for recurrent weights exploit the fact that orthogonal matrices preserve vector length, which stops signals growing or shrinking as they pass through.