Safe Reinforcement Learning

Safe RL chases reward while respecting a hard cost limit. A constraint says what is allowed, a penalty teaches it, and a shield enforces it.

Safe RL keeps expected cost below a threshold: three steps bank 6.0 reward at cost 2.0, then a shield blocks the move that would break the limit

Why Does This Exist?

Standard RL maximizes expected return and treats everything else as commentary. In simulation that is fine. On a robot, a car, or a trading account, the learning itself can do damage: a walker that finds its joint limits by snapping them, a driver that finds the ditch by entering it, a bot that finds ruin by going broke. The lesson arrives after the harm.

You need a way to say some behaviors are off limits even while learning. That is the constrained MDP, or CMDP (Altman, 1999): maximize reward subject to expected cost below a threshold. Penalties teach the constraint through prices, and trust region methods such as Constrained Policy Optimization keep updates inside it. But prices alone still let a large reward outbid safety, and a safe final policy can still crash during training. So the third piece is a shield that inspects each proposed action and swaps the dangerous ones before they execute. This page covers all three: the constraint, the Lagrangian price, and the shield, with a 3-step toy you can check.

Think of It Like This

A teen driver with dual controls

A teen wants the fastest route to the store. Standard RL hands over the keys and says figure it out, which ends with a ditch. Safe RL puts an instructor in the passenger seat with dual controls.

The instructor sets a rule first, never cross the center line, which is the constraint. Then they add a price, every crossing costs you the car for a week, which is the penalty. And when the teen still steers toward oncoming traffic, the instructor brakes, which is the shield. The teen still chooses the route, but the catastrophe never executes.

Where it stops: an instructor sees danger directly. An RL shield needs a written safety rule plus a model predicting what each action does next. Get either wrong and the brake guards the wrong thing.

How It Actually Works

Start with the standard pieces. State $s$ , action $a$ , reward $R(s, a)$ , discount $\gamma$ in $[0, 1]$ , policy $\pi$ . Safe RL adds a cost function $C(s, a)$ and a threshold $d$ . Cost marks harm: joint stress, lane departure, money lost. The CMDP objective is:

$\max_{\pi} \; \mathbb{E}_{\pi}\left[ \sum_t \gamma^t R(s_t, a_t) \right] \quad \text{subject to} \quad \mathbb{E}_{\pi}\left[ \sum_t \gamma^t C(s_t, a_t) \right] \le d$

In words: earn as much as possible while expected discounted cost stays below $d$ . Here $\mathbb{E}_{\pi}$ averages over the randomness of the policy and the environment.

Prices teach, shields enforce

The Lagrangian turns the constraint into a price with a multiplier $\lambda \ge 0$ :

$R'(s, a) = R(s, a) - \lambda C(s, a)$

The agent maximizes penalized reward $R'$ . When cost runs above $d$ , $\lambda$ rises and unsafe actions look worse; when cost sits below $d$ , $\lambda$ falls. Constrained Policy Optimization extends this idea to trust regions: it takes the largest policy step that still keeps the next cost below $d$ .

Prices alone have a hole. If the reward is large enough it outbids the penalty, so the agent pays the fine and crashes anyway. Shielding closes that hole at execution time: before action $a$ reaches the environment, the shield checks it against the safety rule and substitutes a safe fallback $a_0$ on violation. The environment never sees the dangerous choice.

Worked example

Undiscounted over 3 steps ( $\gamma = 1$ ), threshold $d = 2.0$ . The agent walks a ledge where positions 0, 1 and 2 are safe and 3 is the drop:

Step 1, move to 1: reward $2.0$ , cost $0.0$ .
Step 2, move to 2: reward $2.0$ , cost $0.0$ .
Step 3, lean toward 3: reward $2.0$ , cost $2.0$ .

Totals are reward $2.0 + 2.0 + 2.0 = 6.0$ and cost $0.0 + 0.0 + 2.0 = 2.0$ , exactly at the limit $d$ . That is the number the diagram carries: 6.0 banked, 2.0 spent, nothing left to spend.

Now the policy proposes a dash from 2 to 4 for reward $4.0$ and cost $2.0$ . Accepting would push cost to $4.0$ , double the limit. With $\lambda = 1.0$ the penalized value is $4.0 - 1.0 \times 2.0 = 2.0$ , which still beats staying at $0.0$ , so the price alone says go. The shield says otherwise: $2 + 2 = 4 \ge 3$ breaks the rule, so it substitutes stay (reward $0.0$ , cost $0.0$ ) and totals freeze at $6.0$ and $2.0$ . For the sibling view that plans for model error instead of cost limits, see robust reinforcement learning.

Code

A minimal CMDP ledger plus a post-shield. Totals match the worked example, and the shield stops the dash that the penalty would allow.

1def totals(rewards: list[float], costs: list[float]) -> tuple[float, float]:2    """Undiscounted return and cost over the toy episode."""3    return sum(rewards), sum(costs)4
5
6def shield(proposed: str, x: int) -> str:7    """Post-shield for the ledge: positions 3 and above are the drop."""8    effects = {"creep": 1, "stay": 0, "dash": 2}9    return proposed if x + effects[proposed] < 3 else "stay"10
11
12r, c = totals([2.0, 2.0, 2.0], [0.0, 0.0, 2.0])13print(f"reward {r:.1f} cost {c:.1f} at limit 2.0")14# -> reward 6.0 cost 2.0 at limit 2.015print(f"proposed dash, executed {shield('dash', 2)}")16# -> proposed dash, executed stay17print(f"net of dash with lambda 1.0: {4.0 - 1.0 * 2.0:.1f} still beats stay")18# -> net of dash with lambda 1.0: 2.0 still beats stay

The last line is the point: a penalty of $2.0$ against a reward of $4.0$ still leaves $+2.0$ , so only the shield keeps cost at $2.0$ .

Watch Out For

A penalty the agent gladly pays

The symptom: cost stays above $d$ while reward climbs, and raising $\lambda$ a little changes nothing. The agent has done the math: the unsafe shortcut earns more than the fine costs, so it pays the fine every step.

Treat the price as adaptive, not fixed. Raise $\lambda$ until cost falls below $d$ , or stop pricing and start blocking with a shield for the actions that must never execute. If your unsafe action has net $+2.0$ after the penalty, no fixed fine will stop it.

Safe at convergence, dangerous during training

The symptom: the final policy passes every safety test, but training logs show cost spikes all over the early episodes. The method constrained the destination, not the journey.

Use a shield or a barrier method when hardware or money is on the line, not just a converged guarantee. Check the cost curve across training, not only at the end. If early episodes already exceed $d$ , the guarantee arrived too late.

The Quick Version

Safe RL maximizes reward subject to expected cost below a threshold $d$ , which is the CMDP objective.
The Lagrangian prices violations as $R - \lambda C$ with $\lambda$ rising whenever cost exceeds $d$ .
Prices alone fail when reward outbids the fine, so a shield substitutes a safe fallback before the unsafe action executes.
In the 3-step toy, rewards sum to 6.0 and costs sum to 2.0, exactly at the limit, and the shield blocks the dash that would double cost to 4.0.
With $\lambda = 1.0$ the dash still nets $+2.0$ , which proves the penalty allows what the shield must stop.
Judge safety on the training curve, not only the final policy.