Matrix Operations

Matrices add entry by entry, multiply row by column with inner sizes that must match, and transpose flips rows into columns.

Two 2 by 2 matrices multiply row by column: row 1 of A meets both columns of B to make 19 and 22, row 2 makes 43 and 50, and only matching inner sizes let the product exist

Why Does This Exist?

Every layer of a neural network is a matrix operation. A dense layer multiplies a batch of inputs by a weight matrix, adds a bias, and passes the result on. Data preprocessing centers matrices, attention scores them against their transposes, and gradients flow back through the same three operations. If adding, multiplying, and transposing feel shaky, nothing downstream can feel solid.

Vectors, matrices, and tensors introduces what these objects are. This page is the verbs: the three operations, their shape rules, and the one order trap that bites everyone.

The one idea you need first

A matrix shape like $2 \times 3$ reads rows first: 2 rows, 3 columns. Entry $a_{ij}$ sits in row $i$ , column $j$ . Nearly every error in this subject is a shape error, so read shapes before computing anything.

Think of It Like This

Recipes scaled across kitchens

Think of a matrix as a recipe card: rows are dishes, columns are ingredients. Adding two cards blends two menus dish by dish. Multiplying is fulfilling orders: each row of the first card (a dish's needs) meets each column of the second (a supplier's offering) to produce a table of total costs. Transposing rewrites the card sideways, ingredients as rows, so tomorrow's computation reads naturally.

The analogy's limit is deliberate: real matrices multiply numbers, not intentions, and the row-meets-column rule has no flexibility. A dish with 3 ingredient entries can only meet a supplier listing 3 dishes, which is the inner-size rule with an apron on.

How It Actually Works

Addition: entry by entry, same shape required. Add matching positions:

\begin{bmatrix} 1 & 2 \ 3 & 4 \end{bmatrix} + \begin{bmatrix} 5 & 6 \ 7 & 8 \end{bmatrix} = \begin{bmatrix} 6 & 8 \ 10 & 12 \end{bmatrix}

A $2 \times 2$ plus a $2 \times 3$ is undefined, full stop. (Broadcasting in libraries bends this rule; the math does not.)

Multiplication: row by column, inner sizes must match. For $C = AB$ , entry $c_{ij}$ is row $i$ of $A$ dotted with column $j$ of $B$ . An $(m \times n)$ times an $(n \times p)$ gives $(m \times p)$ : the inner $n$ s must agree and the outer sizes survive.

Worked example

A = \begin{bmatrix} 1 & 2 \ 3 & 4 \end{bmatrix}, \qquad B = \begin{bmatrix} 5 & 6 \ 7 & 8 \end{bmatrix}

Both are $2 \times 2$ , so the product is $2 \times 2$ :

$c_{11} = 1 \times 5 + 2 \times 7 = 5 + 14 = 19$
$c_{12} = 1 \times 6 + 2 \times 8 = 6 + 16 = 22$
$c_{21} = 3 \times 5 + 4 \times 7 = 15 + 28 = 43$
$c_{22} = 3 \times 6 + 4 \times 8 = 18 + 32 = 50$

So $AB = \begin{bmatrix} 19 & 22 \ 43 & 50 \end{bmatrix}$ . Reversing the order gives $BA = \begin{bmatrix} 23 & 34 \ 31 & 46 \end{bmatrix}$ , a different matrix, which proves the headline warning: $AB \neq BA$ in general.

Transpose: flip across the diagonal. Rows become columns, so an $(m \times n)$ becomes $(n \times m)$ :

A^T = \begin{bmatrix} 1 & 3 \ 2 & 4 \end{bmatrix}

Entry $(i, j)$ moves to $(j, i)$ . Transposing twice returns the original: $(A^T)^T = A$ .

Watch Out For

Multiplying matrices whose inner sizes disagree

The symptom is a shape error at 1am: a $(4 \times 3)$ weight meeting a $(4 \times 1)$ input and refusing to multiply. The fix is the outer-inner chant: write both shapes down, check the middle numbers match, and read the answer shape off the outer numbers. Nine times out of ten the fix is a missing transpose on one side, which is exactly why transpose exists.

Assuming AB equals BA

Scalar habits die hard, and matrix multiplication is not commutative. The symptom is a silently wrong result, because $BA$ often has a valid shape and computes without complaint, as the worked example shows with two fully valid but different answers. Gradients, rotations, and projections all depend on the order. When porting a formula, preserve the order exactly; when debugging, swap suspects deliberately and compare.

The Quick Version

Addition is entry by entry and needs identical shapes.
Multiplication is row by column: $(m \times n)$ times $(n \times p)$ gives $(m \times p)$ .
The worked product of the two example matrices is $\begin{bmatrix} 19 & 22 \ 43 & 50 \end{bmatrix}$ .
Transpose flips rows and columns: entry $(i,j)$ moves to $(j,i)$ .
$AB \neq BA$ in general, so operand order is part of the formula.
When shapes refuse to multiply, check the inner sizes, then suspect a missing transpose.