Total Derivative

A partial derivative freezes everything but one input. The total derivative unfreezes them all and reports what actually happens when everything moves.

With t driving both x and y into their product, the x route adds 12 and the y route adds 4, so the total rate 16 dwarfs the frozen partial 3

Why Does This Exist?

Partials lie by omission, politely. $\partial f / \partial x$ tells you what happens when $x$ moves and nothing else does. But in any model worth training, nothing else stays still: $x$ and $y$ are both computed from shared upstream parameters, so nudging one dial turns several gears. Optimizers, sensitivity analyses, and hyperparameter studies all need the honest number, the one counting every gear.

The total derivative is that number. It is the multivariable chain rule viewed as an accounting identity: each dependency is a route, each route contributes, and the total is the sum. Confusing it with a partial is one of the quietest bugs in ML mathematics, because the wrong answer looks exactly like the right one with a smaller value.

The one idea you need first

A partial derivative $\partial f / \partial x$ holds all other direct inputs fixed. The total derivative $df / dt$ holds nothing fixed: it follows $t$ through every variable $t$ touches. Same function, different question. One asks about an isolated nudge, the other about a coordinated move, and only the second describes what happens during training.

Think of It Like This

Rent when both price and flat size change

Your rent is price per square metre times area. The landlord raises the unit price and you also move to a bigger flat in the same month. Asking how much of the bill change came from the price alone is the partial question. Asking what your new bill actually is needs both changes plus their interaction.

The total derivative does that full accounting. It takes the price route (new price on the old area), the area route (old price on the new area), and adds them. Skip either route and your budget is wrong in exactly the direction you ignored. The estimate stops working only for large moves, where the interaction term the linear sum drops starts to matter.

How It Actually Works

For $f(x, y)$ with $x(t)$ and $y(t)$ both functions of $t$ :

\frac{df}{dt} = \frac{\partial f}{\partial x}\frac{dx}{dt} + \frac{\partial f}{\partial y}\frac{dy}{dt}

In plain words: $t$ pushes $f$ through $x$ and through $y$ , and the total push is both pushes added. When only one variable depends on $t$ , the other term is zero and the total collapses to a single product. When neither does, both terms vanish, which is the formal way of saying a constant does not move.

The differential form says the same thing for small steps: $df = f_x\, dx + f_y\, dy$ . Each term pairs a frozen sensitivity with the actual movement of its variable. Gradient descent lives on this identity: the loss change over a step is approximately the gradient dotted with the step, which is the total derivative along the step direction.

Worked example

Take $f(x, y) = xy$ with $x = t^2$ and $y = t + 1$ , at $t = 2$ .

Position first: $x = 4$ , $y = 3$ , $f = 12$ . The frozen partial $\partial f / \partial x = y = 3$ . If you stopped there, you would predict that raising $t$ nudges $f$ gently.

The total says otherwise. Via $x$ : $y \cdot dx/dt = 3 \times 4 = 12$ . Via $y$ : $x \cdot dy/dt = 4 \times 1 = 4$ . Total: $12 + 4 = 16$ , more than five times the partial. Direct check: $f(t) = t^2(t+1) = t^3 + t^2$ , so $f'(t) = 3t^2 + 2t = 12 + 4 = 16$ . The partial missed the $y$ route entirely.

Code

Two routes to the same number: the route-by-route total and the substituted direct derivative.

1def total_rate(t: float) -> float:2    x = t**23    y = t + 14    return y * 2 * t + x * 1  # x route plus y route5
6
7def direct(t: float) -> float:8    return 3 * t**2 + 2 * t  # from f(t) = t^3 + t^29
10
11print(total_rate(2.0))  # -> 16.012print(direct(2.0))      # -> 16.0

Both print 16. The first line inside total_rate is the definition in code: each term is one dependency route, and deleting either term reproduces the classic bug.

Watch Out For

Reporting the partial as the answer in a coupled system

In a network every activation depends on shared weights, so almost no variable moves alone. Quoting $\partial f / \partial x$ where $df / dt$ belongs understates sensitivities, misranks feature importance, and mistunes step sizes. The symptom is a sensitivity analysis that disagrees with finite differences. The fix is mechanical: list every variable the input touches and sum the routes.

Detaching a variable and deleting its route in code

Calling .detach() or stop_gradient on a tensor severs a dependency route: the value flows forward but no gradient flows back through it. Used deliberately it implements fixed targets and alternating optimization. Used accidentally, say on a tensor reused downstream, it silently zeroes a term of the total derivative and the model trains on a censored gradient. If a loss stops responding to a parameter it used to track, check for a stray detach before doubting the architecture.

The Quick Version

The partial freezes every other input; the total follows the parameter through all of them.
Total equals the sum over dependency routes of frozen partial times variable rate.
For $xy$ with both factors driven by $t$ , the total 16 dwarfs the partial 3.
The differential $df = f_x dx + f_y dy$ is the same identity for small steps.
Detaching a tensor in code deletes a route from the total, silently and by design or by accident.