Matrix Calculus

The rules for differentiating through vectors and matrices: which expressions give which shapes, and the three identities that cover most of machine learning.

Shapes flow downhill with the dimensions: a scalar over a vector gives a vector, a vector over a vector gives a matrix, and the gradient always matches its variable

Why Does This Exist?

Scalar calculus differentiates one number by another. Machine learning differentiates scalars by vectors (a loss by weights), vectors by vectors (a layer by its input) and scalars by matrices (a loss by a weight matrix). Each combination has a fixed output shape, and getting the shape wrong is the single most common bug in hand-derived gradients.

Matrix calculus is the bookkeeping that keeps every shape straight, plus a short list of identities that covers nearly everything you will ever differentiate. The gradient page teaches what the object means; the Jacobian and Hessian page teaches the two big matrices. This page is the rulebook connecting formulas to shapes.

The one idea you need first

A partial derivative $\partial f / \partial x$ measures how $f$ changes per unit of $x$ with everything else frozen. Matrix calculus applies that measurement entry by entry, then arranges the results in a fixed layout. The only prerequisite is comfort with that scalar idea, covered in derivatives and partial derivatives.

Think of It Like This

A tax table for every kind of income

A tax office publishes one table per income type: wages use this form, dividends use that one, property a third. You do not re-derive tax law per citizen; you classify the income, then apply the matching table.

Matrix calculus works the same way. Classify the expression (scalar-by-vector? vector-by-vector? trace of a product?) and apply the matching identity. The gradient of $Ax$ with respect to $x$ is $A^T$ in denominator layout, always, no derivation needed at 2am. The skill is classification, not algebra.

The analogy stops where layouts differ: rival textbooks arrange results transposed relative to each other (numerator versus denominator layout), so always check which convention your source uses before copying an identity.

How It Actually Works

Shape rule first: in denominator layout (the ML default), the derivative has the shape of the variable you differentiate by, combined with the shape of what you differentiate. Scalar-by-vector is a vector, vector-by-vector is a matrix, scalar-by-matrix is a matrix of matching shape. The gradient always matches its variable, exactly as the scalar gradient matches its parameters.

The three identities that pay the rent:

Linear form: $\nabla_x (a^T x) = a$ . A dot product differentiates to its constant side.
Quadratic form: $\nabla_x (x^T A x) = (A + A^T)x$ , which simplifies to $2Ax$ when $A$ is symmetric. Covariances and Mahalanobis distances are quadratic forms.
Trace trick: $\frac{\partial}{\partial X}\mathrm{tr}(AXB) = A^T B^T$ . Traces turn matrix products into scalars you can differentiate entry by entry.

Worked example

Differentiate $f(w) = w^T A w$ with $A = \begin{bmatrix} 2 & 1 \\ 1 & 3 \end{bmatrix}$ at $w = [1, 2]$ . $A$ is symmetric, so $\nabla f = 2Aw$ :

Aw = \begin{bmatrix} 2 \cdot 1 + 1 \cdot 2 \\ 1 \cdot 1 + 3 \cdot 2 \end{bmatrix} = \begin{bmatrix} 4 \\ 7 \end{bmatrix}, \quad \nabla f = 2 \begin{bmatrix} 4 \\ 7 \end{bmatrix} = \begin{bmatrix} 8 \\ 14 \end{bmatrix}

Sanity check by expanding: $f(w) = 2w_1^2 + 2w_1 w_2 + 3w_2^2$ , so $\partial f/\partial w_1 = 4w_1 + 2w_2 = 8$ and $\partial f/\partial w_2 = 2w_1 + 6w_2 = 14$ at $(1, 2)$ . Identity and brute force agree at $[8, 14]$ .

Code

The point is the check: confirm an analytic matrix gradient against finite differences before trusting it.

1import numpy as np2
3
4A = np.array([[2.0, 1.0], [1.0, 3.0]])5w = np.array([1.0, 2.0])6analytic = (A + A.T) @ w  # quadratic-form identity7h, num = 1e-5, np.zeros(2)8for i in range(2):9    d = np.zeros(2)10    d[i] = h11    num[i] = ((w + d) @ A @ (w + d) - (w - d) @ A @ (w - d)) / (2 * h)12print(analytic)                    # -> [ 8. 14.]13print(np.allclose(analytic, num))  # -> True

Watch Out For

Copying an identity from the other layout convention

Numerator layout arranges derivatives transposed relative to denominator layout, so the same rule appears with every matrix flipped. An identity copied blindly from a statistics text into ML code transposes silently and the shapes still "work" for square matrices, hiding the bug until results drift. Fix one convention per project and translate on import.

Differentiating with respect to a symmetric matrix naively

If $X$ is constrained symmetric, $X_{ij}$ and $X_{ji}$ are the same variable, so entry-by-entry differentiation double counts the off-diagonals. The fix is to differentiate for general $X$ first, then symmetrise the result: take $G + G^T - \mathrm{diag}(G)$ pattern corrections seriously whenever a covariance or kernel matrix is the variable.

The Quick Version

The gradient always has the shape of the variable it differentiates with respect to.
Three identities cover most of ML: linear forms, quadratic forms and the trace trick.
The worked quadratic gives $[8, 14]$ , confirmed both by identity and by expansion.
Layout conventions transpose everything: never mix numerator and denominator sources.
Symmetric variables need symmetrised gradients, or off-diagonals count twice.