Derivatives and Partial Derivatives

A single number that answers one question: if I nudge this input a little, how much does the output move, and in which direction?

The derivative at x equals 2 is the tangent slope 4 with rise 2 over run 0.5, and a partial derivative is the same slope along one slice while the other input is held still

Why Does This Exist?

A model learns by changing numbers. To change one sensibly you need to know what changing it does, and the raw report that the loss went from $2.31$ to $2.29$ is useless on its own, because you cannot tell whether that was your nudge or the noise in the batch. What you need is the response per unit of change, isolated to the one number you touched.

That quantity is the derivative, and it is the atom of every training algorithm in use today. Learning rate schedules, Adam, weight decay, gradient clipping and LoRA are all strategies for spending derivatives well. If this page is fuzzy, everything built on it stays fuzzy, which is why it is the first page of the band.

There is a second reason it matters practically. A derivative reports sensitivity, not just direction. When one parameter has a derivative of $0.0001$ and another has $400$ , you are looking at the reason a single global learning rate struggles, and the reason adaptive optimisers were invented.

Think of It Like This

The speedometer, not the odometer

Your odometer says you have driven $180$ kilometres. That is the total. It tells you nothing about what is happening right now. You could be parked.

The speedometer is the derivative. It reads the rate at this instant: kilometres per hour, right here. Press the accelerator and it climbs immediately, long before the odometer notices.

Everything about derivatives sits in that distinction. The odometer is the function and the speedometer is its derivative. And the speedometer is local: it reports now and promises nothing about the traffic jam in ten minutes. Derivatives are local too, which is exactly why training takes many small steps instead of one confident leap.

How It Actually Works

Start with the honest definition. The derivative of $f$ at a point $x$ is what the rise-over-run slope settles to as the run shrinks to nothing:

f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}

In plain words: move a tiny step $h$ right, see how much the output changed, and divide by the step size. The limit says to keep shrinking that step until the answer stops changing.

That is a definition, not a method. In practice you use rules, and for machine learning you need surprisingly few. The power rule, $\frac{d}{dx}x^n = nx^{n-1}$ , covers polynomials. Sums differentiate term by term. Constants differentiate to zero. That is enough for this page; products and compositions belong to the chain rule.

Reading the sign and the size

Two pieces of information come out, and they get used differently:

The sign says which way. Positive means the output rises as the input rises, so to reduce the output you decrease the input. This is where training's minus sign comes from.
The size says how much per unit. A derivative of $400$ means a nudge of $0.001$ moves the output by roughly $0.4$ . A derivative of $0.0001$ means that same nudge barely registers.

A derivative of exactly zero is a flat spot, a stationary point. It might be a minimum, a maximum, or a saddle, and the derivative alone cannot tell you which.

Partial derivatives: the same idea, more knobs

Real models have millions of inputs, not one. The extension is almost anticlimactic: differentiate with respect to one variable and treat every other variable as a constant. Nothing else changes.

Write it with a curly $\partial$ instead of a straight $d$ to signal that other variables exist and are being held still:

\frac{\partial f}{\partial x}

Read as the change in $f$ per unit of $x$ , with everything else frozen. The freezing is the whole trick, and it is why the mechanical work is no harder than the single-variable case: a term with no $x$ in it is a constant, and constants differentiate to zero.

Collect one partial derivative per parameter into a single vector and you have a gradient, which is the object optimisers actually consume.

Worked example

Take $f(x) = x^2$ and stand at $x = 2$ .

By the power rule $f'(x) = 2x$ , so $f'(2) = 2 \times 2 = 4$ . The output climbs four times as fast as the input, right there. On the diagram above this is a slope triangle: step $0.5$ right along the tangent and it rises by $2$ , so the slope is $2 \div 0.5 = 4$ . The slope is not a property of the curve as a whole: at $x = 1$ it would be $2$ , and at $x = 0$ it would be $0$ . Local, always.

Now the two-variable case. Take $g(x, y) = x^2 + 3y^2$ at the point $(2, 1)$ :

\frac{\partial g}{\partial x} = 2x = 4, \qquad \frac{\partial g}{\partial y} = 6y = 6

Hand check: for the first, $3y^2$ contains no $x$ , so it is constant and vanishes, leaving $2 \times 2 = 4$ . For the second, $x^2$ is the constant, leaving $6 \times 1 = 6$ . The output is more sensitive to $y$ here than to $x$ , even though we stand further along $x$ . That asymmetry comes from the coefficient $3$ , and it is exactly the kind of imbalance that makes optimisation harder.

Code

The snippet below uses only the standard library. The axis argument is the hold-everything-else-still instruction written out, and both prints match the hand values $4$ and $6$ .

1def g(x: float, y: float) -> float:2    return x**2 + 3 * y**23
4
5def partial(f, x: float, y: float, axis: int, h: float = 1e-5) -> float:6    # Perturb ONE input, hold the other still: the whole definition.7    dx, dy = (h, 0.0) if axis == 0 else (0.0, h)8    return (f(x + dx, y + dy) - f(x - dx, y - dy)) / (2 * h)9
10
11print(round(partial(g, 2.0, 1.0, axis=0), 6))  # -> 4.012print(round(partial(g, 2.0, 1.0, axis=1), 6))  # -> 6.013print([round(partial(g, 2.0, 1.0, axis=a), 6) for a in (0, 1)])  # -> [4.0, 6.0]

Central differences are used rather than the one-sided form because they are accurate to second order in $h$ for the same arithmetic. Both routes agree with $[4, 6]$ derived by hand, which is the point of running it.

Watch Out For

Reading a derivative as a prediction instead of a rate

A derivative of $4$ does not mean the output will increase by $4$ . It means the output increases by about $4$ units per unit of input, for a small enough step.

That word small is load-bearing. With $f(x) = x^2$ at $x = 2$ , the derivative predicts that stepping $1$ right raises the output by $4$ , giving $8$ . The true value is $f(3) = 9$ . The derivative was not wrong. The step was too big for a straight-line approximation of a curve to hold.

This is precisely why training uses a learning rate for small steps rather than jumping to where the slope says the bottom is. A large learning rate bets the local slope holds over a long distance, and on a curved surface that bet loses.

Differentiating at a point where the derivative does not exist

Not every function has a derivative everywhere, and ML is full of the exceptions. ReLU has a corner at zero. The absolute value has one too. The L1 norm has corners along every axis. At a corner the left slope and the right slope disagree, so there is no single answer.

Frameworks do not error on this. They pick a value, often $0$ or $1$ , and carry on, so the failure is silent. Usually it is harmless, because landing exactly on the corner in floating point is rare and the chosen subgradient is legitimate.

Where it bites is when data makes the corner common rather than rare. Feed a batch of exact zeros through ReLU, or apply L1 to an already sparse tensor, and a large fraction of gradients come from convention rather than mathematics. If a model plateaus with suspiciously many exact zeros in its gradients, check this before anything else.

The Quick Version

A derivative is a rate of change: output movement per unit of input movement, measured at one point.
Its sign gives the direction and its size gives the sensitivity. Training uses both.
It is local. It describes the slope right here and promises nothing further away, which is why steps are small.
A partial derivative is the identical idea with other variables held frozen, written with $\partial$ rather than $d$ .
Terms without the variable being differentiated are constants, so they drop to zero. That is most of the work.
At $(2, 1)$ on $x^2 + 3y^2$ the partials are $4$ and $6$ , more sensitive to $y$ despite standing further along $x$ .
One partial derivative per parameter, collected into a vector, is a gradient.
Corners such as ReLU, absolute value and L1 have no derivative. Frameworks silently substitute one.