Matrix

A matrix arranges numbers in $m$ rows by $n$ columns, written $X \in \mathbb{R}^{m \times n}$. Row index first, column index second, always — the reverse convention does not exist and assuming it is a reliable source of bugs.

Two matrices dominate practice. A weight matrix maps one layer's width to the next, so its shape encodes the layer's input and output sizes. A batch matrix holds one example per row, so its first axis is the batch. Every dense layer in a network is a matrix multiplication between those two, which is why the shape rules are worth being fluent in rather than looking up.