Numerical ODE Solvers Compared

Stepping along arrows instead of solving exactly: Euler strides tangentially while Runge-Kutta samples the terrain mid-step for far better accuracy.

One step of size 0.1 on exponential decay shows Euler landing at 0.80 while RK4 lands on the true 0.81873

Why Does This Exist?

Closed-form ODE solutions are the exception, and every time-stepper in ML (optimizers, samplers, simulators, neural ODEs) steps a differential equation with finite strides. Step size buys accuracy at the price of evaluations, and the wrong method at the wrong step oscillates, drifts, or explodes on an equation whose true solution sits calm. Euler is the minimal stride; fourth-order Runge-Kutta is the workhorse; stiffness analysis says when both need replacing by implicit schemes. The equations being stepped are in ODEs; sibling discretizations of derivatives and integrals are in finite differences and numerical integration.

The one idea you need first

From $x_n$ at time $t_n$ , every one-step method forms $x_{n+1} = x_n + h\Phi$ for some slope estimate $\Phi$ built from $f$ evaluations. Local error is the one-step mistake; global error accumulates it over $1/h$ steps, typically one order lower.

Think of It Like This

Hiking blind with a compass

A blindfolded hiker follows a spoken slope rule. Euler strides a full $h$ along the slope felt at the boots: fast, and wrong whenever the hill curves mid-stride. Runge-Kutta scouts: feel the slope at the boots, at the midpoint under that slope, again refined, and at the trial endpoint, then blend the four readings with weights $1, 2, 2, 1$ . The blend cancels the curvature error the single reading eats.

On gentle meadows both arrive fine and Euler is cheaper. On hairpins (stiff decay) the long blind stride overshoots the valley and the next stride overcorrects: oscillation or ejection from strides alone, while the true path sits placid.

Where the analogy stops: real solvers adapt $h$ from error estimates and switch to implicit equations (solve, not stride) when stiffness bites.

How It Actually Works

Euler. $x_{n+1} = x_n + h f(x_n, t_n)$ . One evaluation, local error $O(h^2)$ , global $O(h)$ . Stability on $\dot{x} = \lambda x$ needs $|1 + h\lambda| \le 1$ : for real decay $\lambda < 0$ , $h \le 2/|\lambda|$ .

RK4. With $k_1 = f(x_n)$ , $k_2 = f(x_n + hk_1/2)$ , $k_3 = f(x_n + hk_2/2)$ , $k_4 = f(x_n + hk_3)$ :

x_{n+1} = x_n + \tfrac{h}{6}(k_1 + 2k_2 + 2k_3 + k_4)

Four evaluations, local $O(h^5)$ , global $O(h^4)$ . Larger stability region than Euler but still bounded: stiffness eventually demands implicit methods (backward Euler, trapezoidal, BDF) that solve for $x_{n+1}$ inside $f$ .

Worked example

Decay $\dot{x} = -2x$ from $x(0) = 1$ with $h = 0.1$ . Truth: $e^{-0.2} \approx 0.81873$ .

Euler: $x_1 = 1 + 0.1(-2) = 0.80$ . Error $0.0187$ .

RK4: $k_1 = -2$ , $k_2 = -2(1 + 0.05(-2)) = -2(0.90) = -1.8$ , $k_3 = -2(1 + 0.05(-1.8)) = -2(0.91) = -1.82$ , $k_4 = -2(1 + 0.1(-1.82)) = -2(0.818) = -1.636$ . Then $x_1 = 1 + (0.1/6)(-2 - 3.6 - 3.64 - 1.636) = 1 - 0.18127 = 0.81873$ . Error below $10^{-5}$ : four evaluations bought four orders of accuracy, matching truth to the digits shown.

Watch Out For

Stepping a stiff decay with an explicit stride

Rates $1$ and $1000$ together cap explicit steps at $2/1000$ long after the fast mode dies: millions of steps to track a flat line, and any bigger step oscillates then explodes while the ODE itself rests. Error control cannot fix a stability boundary.

Symptom: step adapters shrink to nothing on smooth problems, or solutions blow up at steps that "should" be fine. Fix: detect stiffness from Jacobian spread or step-size collapse, then switch to implicit, exponential, or Rosenbrock methods with adaptive order.

Quoting local order as global accuracy

RK4's $O(h^5)$ per step becomes $O(h^4)$ globally after $1/h$ steps accumulate; Euler's $O(h^2)$ becomes $O(h)$ . Halving $h$ quarters Euler error but divides RK4 error by sixteen, while doubling cost in both.

Symptom: expected digits never arrive (order quoted wrong), or refinement studies confuse pre-asymptotic behaviour with the rate. Fix: verify with a refinement ladder ( $h$ , $h/2$ , $h/4$ ) on the actual equation, and budget evaluations by global order: RK4 costs 4 per step but needs far fewer steps.

The Quick Version

Euler strides tangentially: 1 evaluation, global $O(h)$ , narrow stability.
RK4 blends 4 slope samples: global $O(h^4)$ , far wider accuracy per step.
On the worked decay with $h = 0.1$ , Euler gives 0.80 while RK4 hits 0.81873.
Halving $h$ quarters Euler error and sixteenths RK4 error.
Stiff systems need implicit methods; verify rates with refinement, not slogans.