Adversarial Attacks on RL Policies

A sticker-sized change to what the agent sees can flip what it does. Adversarial attacks on RL policies craft tiny observation changes that push a trained agent into the wrong action.

A clean observation leads the policy to action 0, while the same observation with an epsilon 0.06 FGSM perturbation flips it to action 1.

Why Does This Exist?

Agents that play video games from raw pixels look unbeatable until you nudge a few pixels. In 2017, Huang and coauthors showed that the same gradient tricks that fool image classifiers also fool trained game policies: a change so small you can't see it can tank a Deep Q-Networks agent's score or steer it into its worst action. That stops being a party trick the moment a policy leaves the lab. A delivery robot that reads a doctored sign, or a trading agent fed slightly shifted prices, fails in exactly this way.

This page is about attackers, not accidents. Safe RL asks how to respect constraints when nothing is out to get you. Here you learn how a policy gets pushed off course on purpose: perturbed observations at test time, strikes timed for the moments that count, and poisoned signals during training.

You need one prerequisite idea first. A value-based agent scores every action in a state with Q-values and picks the largest one. If pixels drive those scores, pixels can rig the vote. The full treatment lives in deep reinforcement learning.

Think of It Like This

The doctored stop sign

Researchers once stuck a few small patches on a stop sign. You and I still read STOP without thinking, but a vision model read it as a speed limit sign and acted accordingly. The sign is the observation $s$ , the patches are the perturbation $\delta$ capped by a budget $\epsilon$ , the driver is the policy $\pi$ , and the missed stop is the flipped action.

That is a test-time observation attack, and it maps point for point. The attacker never touches the model's weights. They only edit the input, staying inside a budget a human would ignore, and the policy does the rest by trusting its eyes.

Where the analogy stops: a sticker is visible and a single sign can't fight back across time. In RL the attacker can also strike only at the two or three decisive moments of an episode, or poison the training data itself, and each wrong action compounds into the next state.

How It Actually Works

Symbols for this page: $s$ is the observed state (for example, game pixels), $a$ is the action, $\pi(a \mid s)$ is the policy that picks actions, $Q(s, a)$ is the estimated value of each action, $L$ is the attacker's loss, $\delta$ is the perturbation added to the observation, and $\epsilon$ caps how large $\delta$ may be per pixel.

Observation Perturbations and FGSM

The fast gradient sign method (FGSM) builds $\delta$ in one step:

$s_{adv} = s + \epsilon \cdot \mathrm{sign}(\nabla_s L)$

Read it inside out. $\nabla_s L$ is the gradient of the attacker's loss with respect to the input pixels: it says which direction each pixel should move to help the attacker. $\mathrm{sign}(\cdot)$ keeps only the direction ( $+1$ or $-1$ per pixel), and $\epsilon$ sets the step size. Every pixel moves at most $\epsilon$ , so the change stays invisible while the logits shift enough to flip the argmax.

For a classifier the loss pushes away from the true label. For a policy, Huang and coauthors instead push away from the action the agent currently prefers, for instance by maximizing the gap between a target action's Q-value and the best Q-value. Same formula, new victim.

Worked Example: Epsilon 0.06 Flips the Action

Take a tiny two-action linear policy with weights $w_0 = [1, -1]$ for action 0 and $w_1 = [-1, 1]$ for action 1. Logits are $z_0 = w_0 \cdot s$ and $z_1 = w_1 \cdot s$ , and the agent picks the larger one.

Clean observation $s = [0.60, 0.50]$ :

$z_0 = 0.60 - 0.50 = 0.10, \quad z_1 = -0.60 + 0.50 = -0.10$

Softmax turns those into probabilities. Exponentiating gives $[1.1052, 0.9048]$ , which sums to $2.01$ , so $p = [0.5498, 0.4502]$ . Action 0 wins with about 55 percent.

Now the attacker wants action 1 and uses loss $L = z_1 - z_0$ . Its gradient is $w_1 - w_0 = [-2, 2]$ , whose sign is $[-1, +1]$ . With budget $\epsilon = 0.06$ :

$\delta = 0.06 \cdot [-1, +1] = [-0.06, +0.06], \quad s_{adv} = [0.54, 0.56]$

Fresh logits:

$z_0 = 0.54 - 0.56 = -0.02, \quad z_1 = -0.54 + 0.56 = 0.02$

Exponentiating gives $[0.9802, 1.0202]$ , which sums to $2.0004$ , so $p = [0.49, 0.51]$ . Action 1 now wins. Each input moved only $0.06$ , yet the choice flipped.

Notice why it worked. The clean gap was $0.10 - (-0.10) = 0.20$ . The attack moved the gap by $(w_1 - w_0) \cdot \delta = (-2)(-0.06) + (2)(0.06) = 0.24$ , which beats $0.20$ . The attacker never rewrote the input. They pushed along the gradient just far enough to close the gap.

Striking Only When It Counts

Perturbing every frame is noisy and easy to spot, so Lin and coauthors introduced strategically timed attacks: hit only the few steps where the agent's preference is strong but brittle, and leave the rest untouched. The attacker scores each step by how much the outcome hinges on it, spends the perturbation budget there, and gets most of the damage for a fraction of the tampered frames. Defenses that scan for constant noise miss this completely, because nearly the whole episode looks clean.

Poisoning Training Instead of Testing

Test-time attacks edit what the agent sees. Training-time attacks edit what it learns from. Flipping a fraction of rewards, relabeling which action looked good, or slipping a trigger pattern into some training states plants a backdoor: the policy behaves normally until the trigger appears, then misbehaves on cue. This is why locked datasets, reward sanity checks, and provenance for demonstrations matter. A policy is only as honest as its training signal.

The standard defense is adversarial training: perturb observations during training so the policy learns scores that hold up under attack, then verify with a held-out attacker the training never saw. Constraint-side safety is the sibling topic in Safe RL.

Code

A short check of the worked example above, with type hints throughout:

1import math2
3
4def softmax(z: list[float]) -> list[float]:5    exps = [math.exp(v) for v in z]6    total = sum(exps)7    return [e / total for e in exps]8
9
10def logits(state: list[float]) -> list[float]:11    w0 = [1.0, -1.0]12    w1 = [-1.0, 1.0]13    z0 = w0[0] * state[0] + w0[1] * state[1]14    z1 = w1[0] * state[0] + w1[1] * state[1]15    return [z0, z1]16
17
18s = [0.60, 0.50]19eps = 0.0620
21clean_p = softmax(logits(s))22print([round(p, 4) for p in clean_p])23# -> [0.5498, 0.4502]24
25grad_sign = [-1.0, 1.0]26s_adv = [s[i] + eps * grad_sign[i] for i in range(2)]27print([round(v, 2) for v in s_adv])28# -> [0.54, 0.56]29
30adv_p = softmax(logits(s_adv))31print([round(p, 4) for p in adv_p])32# -> [0.49, 0.51]

1# -> expected output:2[0.5498, 0.4502]3[0.54, 0.56]4[0.49, 0.51]

Watch Out For

Grading the agent in a clean room only

Your policy scores perfectly on the untouched environment, so you ship it. In deployment a little sensor noise, compression artifacts, or a deliberate nudge flips exactly the close calls you never tested, and the failure lands on the highest stakes states.

Test under fire before you trust a score. Evaluate with FGSM and a multi-step attacker at a stated epsilon budget, plus plain corruptions like blur and quantization, and report those numbers next to the clean score.

Gradient masking that feels like safety

You harden the model, the attacker's gradient goes flat or noisy, their FGSM fails, and you declare victory. Then a black-box attacker who never needed your gradients, or the same attack retrained against your new weights, walks straight through. Hidden gradients are obscurity, not safety.

Verify with attackers that don't use your model's gradients: transfer attacks from a surrogate policy, score-based queries, and random search inside the epsilon ball. Pair that with adversarial training instead of gradient shaping.

The Quick Version

Tiny input changes flip RL actions: FGSM adds $\epsilon \cdot \mathrm{sign}(\nabla_s L)$ to the observation and the policy picks a different argmax.
A worked flip you can check: clean logits $[0.10, -0.10]$ pick action 0 at 55 percent, and an $\epsilon = 0.06$ push gives $[-0.02, 0.02]$ , which picks action 1 at 51 percent.
Timed attacks perturb only the decisive steps, so scanning for constant noise misses them.
Poisoned rewards or doctored training states plant backdoors that behave normally until a trigger appears.
Clean-environment scores overstate safety, so report attacked scores at a stated epsilon next to them.
Gradient masking hides the attack without removing it, so test with black-box and transfer attackers too.