Frechet and Gateaux Derivatives

Gateaux differentiates along each straight direction; Frechet demands one linear map that works uniformly in all directions at once. Uniformity is what the chain rule needs.

The curve rises through the point with slope 10 while the tangent line touches once and the gap shrinks quadratically

Why Does This Exist?

Optimizing over functions (neural operators, variational problems, PDE-constrained design) needs calculus beyond $\mathbb{R}^n$ . Directional slopes are easy to define anywhere, but backpropagation, implicit differentiation and Newton steps need a single linear map approximating the function uniformly. Confusing the two produces chain rules that fail and gradients that mislead.

This page separates Gateaux (directional) from Frechet (uniform) and checks both on one bowl. Single-variable rates are in derivatives and partial derivatives; the finite-dimensional uniform map is the total derivative.

The one idea you need first

A directional derivative fixes a direction $h$ and differentiates $t \mapsto f(x + th)$ at $t = 0$ . A bounded linear map $L$ satisfies $L(\alpha u + \beta v) = \alpha Lu + \beta Lv$ with $\|Lu\| \le C\|u\|$ . Frechet asks one such $L$ to fit all directions uniformly.

Think of It Like This

Surveying a hill versus paving it flat

Gateaux walks straight lines through your position and reports each trail's slope: north trail 4, east trail 6, diagonal trail 10. Useful, but trails can disagree wildly (one cliff, one meadow) with no single plane fitting them.

Frechet lays one tangent plane touching the hill uniformly: every direction's slope reads off the same plane, with errors shrinking faster than distance in all directions at once. Where it stops: Gateaux-linearity in each direction separately never builds the plane. Uniformity across directions is the extra purchase.

How It Actually Works

$f$ is Gateaux differentiable at $x$ when every directional limit exists:

d f(x; h) = \lim_{t \to 0} \frac{f(x + th) - f(x)}{t}

In plain words: freeze a direction $h$ , walk it, read the slope. It is Frechet differentiable when one bounded linear $Df(x)$ satisfies:

f(x + h) - f(x) - Df(x)h = o(\|h\|)

uniformly over directions: the remainder vanishes faster than $\|h\|$ no matter which way $h$ points. Frechet implies Gateaux with $df(x;h) = Df(x)h$ , continuous Gateaux on a neighborhood implies Frechet, but bare Gateaux alone never chains or optimizes safely.

Worked example

$f(x, y) = x^2 + 3y^2$ at $(2, 1)$ along $h = (1, 1)$ . Gateaux: $\phi(t) = f(2+t, 1+t) = (2+t)^2 + 3(1+t)^2 = 7 + 10t + 4t^2$ , so $\phi'(0) = 10$ . Directional slope $10$ .

Frechet candidate $Df = [4, 6]$ (the gradient). Remainder: $f(2+t, 1+t) - f(2,1) - [4,6]\cdot(t,t) = (7 + 10t + 4t^2) - 7 - 10t = 4t^2$ . With $\|h\| = |t|\sqrt{2}$ : $4t^2 / (|t|\sqrt{2}) = 2\sqrt{2}|t| \to 0$ uniformly (one direction shown; the quadratic bound holds for all). Frechet holds with the same map every direction shares.

Code

Directional slope by finite differences matches the gradient dot direction, with quadratic remainder decay.

1import numpy as np2
3
4def gateaux(f: object, x: np.ndarray, h: np.ndarray, t: float = 1e-6) -> float:5    return float((f(x + t * h) - f(x - t * h)) / (2 * t))  # central difference6
7
8f = lambda v: v[0] ** 2 + 3 * v[1] ** 29x = np.array([2.0, 1.0])10h = np.array([1.0, 1.0])11print(round(gateaux(f, x, h), 3))  # -> 10.012print(round(float(f(x + 0.01 * h) - f(x) - 10 * 0.01), 6))  # -> 0.0004

Slope $10.0$ ; remainder at $t = 0.01$ is $4 \times 10^{-4} = 0.0004$ : quadratic, exactly as derived.

Watch Out For

Chaining Gateaux derivatives like Frechet ones

Symptom: backprop through a merely Gateaux map silently produces wrong gradients on some directions. Fix: verify Frechet (uniform remainder) before any chain rule; Gateaux linearity per direction is never enough.

Assuming partials imply differentiability

Symptom: existing partial derivatives quoted as a total derivative at a discontinuity (classic $xy/(x^2+y^2)$ -style traps). Fix: require continuous partials in a neighborhood, or prove the $o(\|h\|)$ remainder directly.

The Quick Version

Gateaux differentiates along each line through the point; Frechet fits one uniform linear map.
$x^2 + 3y^2$ at $(2,1)$ has directional slope $10$ along $(1,1)$ with remainder $4t^2$ .
Frechet implies Gateaux; continuous Gateaux implies Frechet; bare Gateaux implies neither chain rule nor continuity.
Only Frechet derivatives chain, backpropagate and drive Newton steps safely.
Partial derivatives alone never certify differentiability.