Skip to content
AI360XpertBeta

Adversarial Examples: Tiny Noise, Wrong Answer

A shift too small to see can still flip a network's answer. An adversarial example is an input nudged along the model's own gradients until it lands on the wrong side of the boundary.

Pushing each input by 0.10 along the loss gradient drops the true score from 0.50 to 0.32, below the rival 0.40, so the label flips
Pushing each input by 0.10 along the loss gradient drops the true score from 0.50 to 0.32, below the rival 0.40, so the label flips

Why Does This Exist?

A model can ace its test set and still flip its answer on an input you can't tell apart from a normal one. That's not a bug in one architecture. It follows from how classifiers work: they cut a high dimensional space with boundaries, and most inputs sit closer to a boundary than you'd guess. A nudge that is tiny per value can still carry a point across.

The stakes are concrete. A vision system that reads a stop sign correctly in testing can misread one with a few stickers on it. A spam filter or a face check can wave through something it should block. The model isn't broken in the usual sense. It's answering a slightly different question than you think: not "what is this" but "which side of my boundary is this on."

This page covers the attack: how a single gradient step builds the nudge, with a three-number example you can check by hand. The defense lives on its own page: adversarial training mixes these attacks into training so the boundary moves to cover them. A nearby but different idea is differential privacy, which adds noise to protect training data instead of fooling a model.

The one idea you need first: training follows gradients downhill. A gradient is a list of slopes, one per value, pointing where the error climbs fastest, and a loss function turns a bad answer into a number to shrink. An attack reuses both and points them the other way: freeze the weights, then move the input uphill instead.

Think of It Like This

A key filed down by a hair per tooth

You have a key with three teeth, and a lock that reads their heights. Now someone files each tooth by a hair: one a touch shorter, one a touch taller, one a touch shorter. You'd never notice holding it. But the lock doesn't look at the key the way you do. It adds up all three readings, and the total now matches a different door. Same key to your eyes, wrong door to the lock.

Each tooth is an input value. The lock's readings are the model's weights. The filing is the perturbation: small per tooth, aimed with the lock's own blueprint (the gradients) so the total tips over. That's the whole trick. Shifts that mean nothing one by one, chosen so their sum means everything.

Where it stops: a real lock checks each pin against a fixed cut, so errors don't pile up across pins. A neural network sums thousands of small pushes, which is why the total can flip the answer while each push stays invisible. And unlike a burglar with a file, the attacker here aims with the model's own gradients, which is what makes the nudge worst case instead of random.

How It Actually Works

Freeze the weights, move the input. Write L(x)\mathcal{L}(x) for the loss of the correct label on input xx. Training nudges the weights to shrink that number. An attack nudges xx to grow it, with the weights held still.

The one step that builds the nudge

xadv=x+ϵ×sign(∇xL)x_{\text{adv}} = x + \epsilon \times \text{sign}\left(\nabla_{x} \mathcal{L}\right)

In plain words: xx is the input (say, pixel values), L\mathcal{L} is the loss for the correct label, ∇xL\nabla_{x} \mathcal{L} (read "the gradient of the loss with respect to xx") is one slope per input value with the weights frozen, sign\text{sign} keeps only each slope's direction (+1+1 or −1-1), and ϵ\epsilon (read "epsilon") is the budget, the largest change allowed to any single value.

The sign matters more than it looks. The raw gradient would let one steep input hog the budget while flat ones sit out. Taking the sign spends the full ϵ\epsilon on every value, which is the steepest move allowed when size is measured per value (the L∞L_{\infty} budget). Training steps against its gradient to lower the loss. The attacker steps with it to raise the loss of the right answer.

Why small per value still wins

Give each input its own small push and the pushes gang up. For a linear score s=w⋅x+bs = w \cdot x + b (weights ww, bias bb), the change is Δs=w⋅Δx\Delta s = w \cdot \Delta x: each term is one weight times one push. The sign trick picks every push so that every term drags the same way, and the total drop equals ϵ\epsilon times the sum of the weight magnitudes. With thousands of inputs that sum is large even when ϵ\epsilon is small. That's the whole security problem in one sum: per-value invisible, total decisive.

Worked example: three numbers flip a label

Take weights w=[0.6,−0.4,0.8]w = [0.6, -0.4, 0.8], input x=[0.50,0.50,0.50]x = [0.50, 0.50, 0.50], bias b=0b = 0. The true score is s=0.6×0.50−0.4×0.50+0.8×0.50=0.30−0.20+0.40=0.50s = 0.6 \times 0.50 - 0.4 \times 0.50 + 0.8 \times 0.50 = 0.30 - 0.20 + 0.40 = 0.50. Say the rival class scores a fixed r=0.40r = 0.40, so the model is right: 0.500.50 beats 0.400.40.

Attack with a loss that falls as the true score rises, L=−s\mathcal{L} = -s. Its gradient over the input is ∇xL=−w=[−0.6,+0.4,−0.8]\nabla_{x} \mathcal{L} = -w = [-0.6, +0.4, -0.8], with signs [−1,+1,−1][-1, +1, -1]. With budget ϵ=0.10\epsilon = 0.10, the push is [−0.10,+0.10,−0.10][-0.10, +0.10, -0.10], giving xadv=[0.40,0.60,0.40]x_{\text{adv}} = [0.40, 0.60, 0.40]. No value moved more than 0.100.10, yet:

sadv=0.6×0.40−0.4×0.60+0.8×0.40=(0.24−0.24+0.32)=0.32s_{\text{adv}} = 0.6 \times 0.40 - 0.4 \times 0.60 + 0.8 \times 0.40 = \left(0.24 - 0.24 + 0.32\right) = 0.32

The true score drops by 0.180.18 (exactly ϵ\epsilon times 0.6+0.4+0.8=1.80.6 + 0.4 + 0.8 = 1.8), landing at 0.320.32, under the rival 0.400.40. Label flipped. The diagram below draws these same numbers: three bars shift, the true bar sinks past the rival line.

Whoever holds the gradients holds the attack

A white-box attacker has the weights and computes the exact gradient above. A black-box attacker gets only inputs and outputs, yet the attack often still lands: examples built on a substitute model tend to fool the target too, a property called transferability. So hiding your weights is not a defense.

Two trade-offs to carry with you. One step (FGSM) is cheap and often enough, but iterating smaller steps flips more inputs at a higher compute cost. And the attack assumes the model responds smoothly near xx and the attacker can feed chosen inputs. It fails where inputs pass through strict checks or heavy quantization, and it stops working once the boundary itself is retrained on attacks, which is the job of adversarial training.

Code

The point of this snippet is the check: it replays the worked example and prints the two scores, so you can confirm the flip runs as claimed.

import numpy as np

def fgsm_step(x: np.ndarray, grad: np.ndarray, eps: float) -> np.ndarray:    return x + eps * np.sign(grad)

w: np.ndarray = np.array([0.6, -0.4, 0.8])x: np.ndarray = np.array([0.50, 0.50, 0.50])grad_loss: np.ndarray = -w  # loss falls as the true score risesx_adv: np.ndarray = fgsm_step(x, grad_loss, 0.10)print(x_adv)  # -> [0.4 0.6 0.4]print(round(float(w @ x), 2))  # -> 0.5print(round(float(w @ x_adv), 2))  # -> 0.32

The printout matches the hand computation: the shifted input is [0.40,0.60,0.40][0.40, 0.60, 0.40], and the scores are 0.500.50 then 0.320.32. The rival at 0.400.40 sits between them, which is the flip.

Watch Out For

Random noise passes, so the model must be safe

Gaussian noise at the same budget changes nothing, the demo looks great, and the team ships. Then a worst-case nudge inside that identical budget flips the model. Random noise tests the average direction. Attacks use the worst one, and the gap between them is the whole story.

Fix: test with crafted nudges at a stated ϵ\epsilon, not random static. Quote the budget next to every claim, and treat any safety report without one as untested.

Blocking the gradient looks like a fix

A non-differentiable filter in front of the model stops white-box FGSM cold, and every demo passes. Then the attacker trains a substitute, crafts the nudge there, and it transfers straight through the filter. The symptom is a model that beats the attack you test and falls to the one you didn't.

Fix: evaluate transfer attacks from a substitute model before claiming anything. For an actual fix, move the boundary by training on the attacks themselves, in adversarial training, instead of hiding the slope.

The Quick Version

  • An adversarial example is a normal input plus a worst-case nudge inside a small budget, built from the loss gradient, not random noise.
  • FGSM takes one step: the input plus epsilon times the sign of the gradient, so every value moves by exactly epsilon.
  • In the worked toy, shifts of 0.10 drag the true score from 0.50 to 0.32, under the rival 0.40, and the label flips.
  • Nudges built on one model often fool another, so hiding weights or blocking gradients is not a defense.
  • The standard answer is training on the attacks themselves, which costs extra compute and some clean accuracy.