Hadamard Product

Multiplication done seat by seat: each entry times its counterpart, no mixing. Masks, gates and attention weights all multiply this way.

Hadamard pairs seats with seats so 1 2 3 4 with 5 6 7 8 gives 5 12 21 32, while matrix multiplication would mix every row with every column

Why Does This Exist?

Two multiplications share the word "matrix" and do opposite things. The ordinary product mixes: each output entry blends a whole row with a whole column. The Hadamard product $A \odot B$ never mixes: entry $(i, j)$ of the output is just $a_{ij} \cdot b_{ij}$ , seat by seat. Same shapes in, same shape out.

That locality is exactly what gating needs. An attention mask zeroes forbidden positions without touching the rest. Dropout scales kept activations without redistributing anything. LSTM gates throttle each coordinate independently. Whenever the operation should respect positions rather than combine them, the seat-by-seat product is the tool, and reaching for the mixing product instead is a shape-compatible bug.

The one idea you need first

Ordinary matrix multiplication $(AB)_{ij} = \sum_k a_{ik} b_{kj}$ blends rows with columns and changes shapes. Hadamard needs identical shapes and preserves them. The contrast in matrix multiplication is worth one fresh look before the example below.

Think of It Like This

Two transparencies stacked on a projector

Print two patterns on transparencies and stack them on a projector. Each point on the screen shows the product of the two inks at exactly that point: dark over dark stays dark, clear over anything passes it through. No light leaks sideways from one point to another.

Matrix multiplication is the opposite machine: a blender that pours every row into every column. The transparency stack is local and shape-preserving; the blender is global and shape-changing. Masks and gates are transparencies, which is why they multiply entry by entry.

The analogy stops at differentiability: transparencies just sit there, while Hadamard layers pass gradients seat by seat too, which is what makes masking trainable.

How It Actually Works

For same-shaped $A$ and $B$ :

(A \odot B)_{ij} = a_{ij} \cdot b_{ij}

Three facts worth carrying:

Masking: multiplying by a 0/1 matrix keeps or kills each entry independently. Causal masks, padding masks, dropout masks.
Schur product theorem: the Hadamard product of two positive semidefinite matrices stays positive semidefinite. Kernel engineers rely on this when multiplying kernels entry by entry.
Gradient locality: $\partial (A \odot B) / \partial A = B$ entry by entry. Backprop through a mask scales each gradient by its own mask value, nothing redistributes.

Worked example

Pair $\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$ with $\begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix}$ seat by seat:

A \odot B = \begin{bmatrix} 1 \cdot 5 & 2 \cdot 6 \\ 3 \cdot 7 & 4 \cdot 8 \end{bmatrix} = \begin{bmatrix} 5 & 12 \\ 21 & 32 \end{bmatrix}

Compare with the mixing product $AB = [[19, 22], [43, 50]]$ : entry $(1,1)$ blends $1 \cdot 5 + 2 \cdot 7 = 19$ instead of just $5$ . Same inputs, 5 versus 19. Any code review that cannot say which product a line intends has found a bug-or-luck situation.

Code

The point is the check: confirm entry-by-entry behaviour and contrast it with the mixing product on the same inputs.

1import numpy as np2
3
4A = np.array([[1, 2], [3, 4]])5B = np.array([[5, 6], [7, 8]])6print((A * B).tolist())  # -> [[5, 12], [21, 32]]7print((A @ B).tolist())  # -> [[19, 22], [43, 50]]

A * B is the Hadamard product in NumPy; A @ B mixes. The two outputs share no entries, which is why confusing them never "almost works".

Watch Out For

Broadcasting silently reshapes a Hadamard product

NumPy broadcasts (4,) against (4, 3) without complaint, pairing each row with the same vector. Intended as per-row scaling, fine; intended as same-shape masking, silently wrong. Assert shapes match exactly before any masking multiply, because broadcasting turns typos into passing tests.

Writing * in math and @ in code, or the reverse

Mathematics writes the mixing product as juxtaposition $AB$ and marks Hadamard with $\odot$ ; NumPy writes mixing as @ and Hadamard as *. Translating notation by shape of symbol rather than meaning flips the operation. When porting formulas, map the operation (mixing vs seat-by-seat), never the glyph.

The Quick Version

Hadamard multiplies seat by seat: $(A \odot B)_{ij} = a_{ij} b_{ij}$ , shapes unchanged.
The worked pair gives [[5, 12], [21, 32]] versus [[19, 22], [43, 50]] for the mixing product.
Masks, gates and dropout are Hadamard: locality is the feature, not a limitation.
Gradients flow seat by seat too, scaled by each entry's own mask value.
In NumPy * is Hadamard and @ mixes; assert shapes before masking.