Shielding

Why Does This Exist?

Standard RL learns by trying things, and some things should never be tried even once. A robot arm that finds its joint limits by snapping them, or a trading bot that finds ruin by going broke, has learned a lesson you cannot afford.

Safe RL frames this problem with costs and constraints: maximize reward while keeping expected cost below a threshold. That framing usually promises a safe policy at the end of training. But during training the agent still walks off the cliff. It has to, because that is how it learns where the cliff is.

Shielding fixes training itself. A shield sits between the policy and the environment, inspects every proposed action, and replaces the dangerous ones before they execute. The agent never experiences the catastrophe, so it never needs to learn the penalty for one.

This page covers the shield mechanism: pre-shield versus post-shield, the written safety spec, minimal interference, and why this is not reward shaping. The broader costs-and-constraints picture lives on the Safe RL page.

Think of It Like This

How It Actually Works

The policy proposes an action $a$ in state $s$. The shield computes a verdict. The environment only ever sees the shield's output:

$\text{shield}(s, a) = \begin{cases} a & \text{if } a \text{ keeps the spec true} \\ a_0 & \text{otherwise} \end{cases}$

where $a_0$ is a known safe fallback action and $s$ is the current state. Learning continues as normal on the executed action, but the catastrophe never executes.

Pre-shield versus post-shield

A pre-shield filters the action set before the policy chooses. In state $s$ it hands the policy only the safe subset, so the policy never considers the cliff. This is cheap at decision time but needs the safe set computed upfront for every state.

A post-shield monitors after the policy proposes. The policy ranks all four actions freely, proposes its favorite, and the shield overrides only on violation. This works with any policy, including one you already trained, at the cost of one safety check per step. The worked example below uses a post-shield.

The safety spec in LTL

The rule is written in linear temporal logic (LTL), a language for statements about behavior over time. Our toy spec is:

$\varphi = G(x < 3)$

Read it as: at every future moment ($G$, always), the position $x$ stays left of 3. Given this spec plus a simple model of how actions move the agent, the shield is synthesized from a safety game, and it is correct by construction for as long as the model matches reality.

Worked example: four actions on a ledge

The agent stands at $x = 2$. Positions 0, 1 and 2 are safe; anything at 3 or beyond is the cliff. The fallback is $a_0 = \text{stay}$. The shield evaluates each proposal by adding the move effect and testing $\varphi$:

• left: $2 + (-1) = 1$, and $1 < 3$, so ALLOW. The shield passes it through and the agent steps to 1.
• right: $2 + 1 = 3$, and $3 < 3$ is false, so BLOCK. The shield substitutes stay and the agent stays at 2.
• stay: $2 + 0 = 2$, and $2 < 3$, so ALLOW. It passes through untouched.
• jump (+2): $2 + 2 = 4$, and $4 < 3$ is false, so BLOCK. The shield substitutes stay and the agent stays at 2.

Two of the four proposals pass through exactly as proposed. That is minimal interference: the shield changes the minimum needed to keep $\varphi$ true and leaves everything else alone. Note what never changed: the reward function. No penalty was added, no bonus removed.

Why this is not reward shaping

Reward shaping edits the reward signal, say $-10$ for visiting the cliff, and hopes the policy learns to avoid it. But a policy maximizing shaped reward can still pick the cliff whenever the gain exceeds 10, and it must visit the cliff during training to learn the penalty exists. The guarantee is soft and arrives late.

A shield removes the choice instead of repricing it. Even with garbage Q-values on day one of training, the cliff action cannot execute, because the environment never receives it. The price is that you must write the spec and supply a model of action effects. Shaping needs neither, which is exactly why it guarantees neither.

Code

A post-shield for the ledge toy. The spec check is one comparison, and the fallback is always stay:

1def shield(proposed: str, x: int) -> str:2    """Post-shield for the ledge toy: spec G(x < 3), fallback stay."""3    effects = {"left": -1, "right": 1, "stay": 0, "jump": 2}4    if x + effects[proposed] < 3:5        return proposed6    return "stay"7
8
9x = 210for a in ["left", "right", "stay", "jump"]:11    print(f"proposed {a}, executed {shield(a, x)}")12    # -> proposed left, executed left13    # -> proposed right, executed stay14    # -> proposed stay, executed stay15    # -> proposed jump, executed stay

Watch Out For

The Quick Version

• A shield sits between policy and environment and replaces actions that would break a written safety spec before they execute.
• A pre-shield filters the action set before choosing, while a post-shield overrides bad proposals after the policy picks.
• The spec is written in LTL, such as $G(x < 3)$, and the shield intervenes only when a proposal would violate it.
• In the ledge toy at $x = 2$, left and stay pass through while right and jump are swapped for stay, so 2 of 4 moves are stopped.
• Reward shaping reprices danger with penalties but still lets the agent visit it, while shielding removes the choice and holds during training.

What to Read Next

• Safe Reinforcement Learning
• Q-Learning
• Temporal Difference Learning