Lyapunov-based Safe RL

Lyapunov safe RL gives the agent an energy meter that must drop on every step, so only actions that move toward the goal count as safe.

Lyapunov level sets shrink toward the goal and only the action that lowers L(x) from 4.0 to 2.25 is accepted while the move to 5.29 is blocked.

Why Does This Exist?

Average constraint methods say the agent is safe if its total cost stays below a threshold. That lets a single step spike into danger as long as the average still looks fine. A robot arm can snap a joint once and the average will forgive it. A trading policy can blow past a risk limit for one hour and still pass the monthly audit.

Safe RL introduces that average constraint view with Constrained MDPs. This page adds the stricter per step certificate used when you need stability, not just a good average. You will learn the Lyapunov decrease rule, the Lyapunov critic that enforces it, and the feasible set it protects. If you want the broad safety landscape first, read Safe RL. If you want planning with a learned model, see model-based RL.

Think of It Like This

A marble in a mixing bowl

Imagine a marble rolling inside a smooth mixing bowl. Height is energy. The rim is the safety boundary and the bottom is your goal.

You allow only pushes that leave the marble lower than it started. A gentle push toward the center passes because height drops from 4.0 to 2.25. A push up the wall fails because height climbs from 4.0 to 5.29, even though the marble is still inside the bowl. Staying inside is not enough. It must settle lower each time.

Where the analogy stops: a real bowl gets gravity for free. In RL you must learn the energy function from data, and a bad estimate lets unsafe actions slip through.

How It Actually Works

A Lyapunov function $L$ is a learned energy meter over states. Call the current state $x$ , the action $a$ , and the next state $x'$ . Call the stage cost $c(x)$ , a non negative number that is zero only at the goal. Call $\alpha$ a small positive margin rate. A valid $L$ satisfies $L(0) = 0$ at the goal and $L(x) > 0$ everywhere else.

The decrease condition

The policy is certified stable when every step obeys the discrete decrease rule:

$L(x') - L(x) \leq -\alpha \cdot c(x)$

In words: the next energy must sit below the current energy by at least a small margin. For the simple version with $\alpha \cdot c(x)$ folded into a strict drop, read it as $L(x') < L(x)$ . If an action breaks the inequality, the safety filter blocks it and picks the closest passing action instead. That is the whole certificate. No drop, no step.

The Lyapunov critic and the feasible set

In practice $L$ is a neural critic $L_\theta$ trained much like a value function, using transitions $(x, a, x', c)$ to predict long term cost to goal. The policy $\pi$ then optimizes reward subject to the critic constraint:

$\mathbb{E}[L_\theta(x') \mid x, a] \leq L_\theta(x) - \alpha \cdot c(x)$

Here $\mathbb{E}$ means the average over stochastic next states, $x'$ is the random next state, and $a$ comes from $\pi$ . The feasible set is one sublevel of $L$ :

$\mathcal{S}_d = \{x \mid L(x) \leq d\}$

Pick the threshold $d$ so the starting state sits inside. The decrease rule makes $\mathcal{S}_d$ invariant. Once inside, a certified policy can never climb out, because climbing would need $L$ to rise. This is stronger than trust region steps in TRPO, which limit how far the policy moves but do not certify each state stays inside an energy level.

Worked example with L(x) = x squared in 1D

Take the 1D candidate $L(x) = x^2$ . The goal is $x = 0$ with $L(0) = 0$ . Set the safe threshold $d = 9$ , so the safe set is $|x| \leq 3$ . Start at $x = 2.0$ :

$L(2.0) = 2.0^2 = 4.0$

Two candidate actions propose two next states.

Safe candidate: $x' = 1.5$ .

$L(1.5) = 1.5^2 = 2.25$ $\Delta L = 2.25 - 4.0 = -1.75$

The change is negative, so the step passes. With $c(x) = x^2 = 4.0$ and $\alpha = 0.2$ , the required drop is $-0.8$ . The actual $-1.75$ clears it with room to spare.

Risky candidate: $x' = 2.3$ .

$L(2.3) = 2.3^2 = 5.29$ $\Delta L = 5.29 - 4.0 = +1.29$

The change is positive, so the filter blocks it. Notice the trap: $5.29 \leq 9$ , so the state is still inside the safe set. Inside is not enough. Without a drop, the policy could drift uphill until it exits. The diagram shows this as two arrows from the $L = 4.0$ ring: the inward mint arrow lands on $L = 2.25$ and the outward amber arrow is stopped before $L = 5.29$ .

One more step completes the picture. From $x = 1.5$ , pick $x' = 1.0$ :

$L(1.0) = 1.0$ $\Delta L = 1.0 - 2.25 = -1.25$

Again negative, again accepted. Repeated drops steer $x$ to $0$ and $L$ to $0$ .

Code

The snippet below is the exact check the filter runs. It uses the same numbers as the worked example.

1def lyapunov(x: float) -> float:2    """Energy candidate L(x) = x squared."""3    return x * x4
5
6def passes_decrease(x: float, x_next: float, alpha: float = 0.2) -> bool:7    """Return True only when the Lyapunov value shrinks enough."""8    stage_cost: float = x * x9    return lyapunov(x_next) <= lyapunov(x) - alpha * stage_cost10
11
12if __name__ == "__main__":13    start = 2.014    safe_next = 1.515    risky_next = 2.316    print(f"L(start) = {lyapunov(start)}")17    print(f"L(safe) = {lyapunov(safe_next)}, passes = {passes_decrease(start, safe_next)}")18    print(f"L(risky) = {lyapunov(risky_next)}, passes = {passes_decrease(start, risky_next)}")

1# -> expected output:2L(start) = 4.03L(safe) = 2.25, passes = True4L(risky) = 5.29, passes = False

Watch Out For

An invalid Lyapunov candidate passes the code but proves nothing

A function with $L(0) \neq 0$ , with negative values, or with flat valleys away from the goal can still shrink on your test batch and fool the filter. Symptom: training looks safe, then the deployed policy rests at a non goal state where $L$ bottoms out above zero.

Fix it by construction. Pin $L(0) = 0$ in code, use a positive head such as a squared norm plus a small quadratic, and probe $L$ on a grid before trusting it. If any $x \neq 0$ gives $L(x) \leq 0$ , reject the candidate.

Checking the continuous slope instead of the next state drop

Learners copy $\dot{L} < 0$ from control textbooks and check gradients in a discrete simulator. Symptom: the gradient points downhill at $x$ , yet the finite step lands uphill at $x'$ , so $L$ rises from $4.0$ to $5.29$ while the slope check smiled and waved it through.

Fix it by always evaluating $L(x') - L(x)$ on the realized next state, averaged over stochastic outcomes. In discrete RL the only decrease that counts is the measured drop.

The Quick Version

Lyapunov safe RL certifies each step with an energy function $L$ that must shrink, so $L(x') < L(x)$ is the gate every action passes through.
The feasible set is one energy level $\{x \mid L(x) \leq d\}$ , and the decrease rule keeps a certified policy inside it for all future steps.
A Lyapunov critic $L_\theta$ learns long term cost to goal, and the policy may only pick actions with expected drop $\mathbb{E}[L(x')] \leq L(x) - \alpha c(x)$ .
In the 1D check with $L(x) = x^2$ , the move $2.0$ to $1.5$ drops energy $4.0$ to $2.25$ and passes, while $2.0$ to $2.3$ lifts energy to $5.29$ and is blocked.
Staying inside the safe set is not enough. An action inside the boundary still fails when it climbs uphill.