Jacobian Matrix

For a function taking $n$ inputs to $m$ outputs, the Jacobian is an $m \times n$ matrix whose entry at row $i$, column $j$ is $\partial \text{out}_i / \partial \text{in}_j$. A gradient is the special case where there is one output, so the Jacobian collapses to a single row.

It is the object the chain rule actually composes for multi-output functions: the Jacobian of a composition is the product of the individual Jacobians. That is what backpropagation is doing at every layer.

The practical detail is that these matrices are never built. A layer mapping 4,096 inputs to 4,096 outputs has a Jacobian with nearly 17 million entries, and storing one per layer is out of the question. Autodiff frameworks instead compute a vector-Jacobian product — the result of multiplying a vector by the Jacobian, obtained directly without forming the matrix. That single optimisation is what makes reverse-mode differentiation practical.