Core ML

Adversarial Examples

By adding invisible, mathematical noise to an input, attackers can trick a highly accurate neural network into making bizarre, confidently incorrect predictions.

An image of a panda is modified with a tiny layer of mathematical noise. To a human, it still looks exactly like a panda, but the neural network confidently classifies it as a gibbon.

Why Does This Exist?

Neural networks do not "see" the world the way humans do. A human looking at a photo of a stop sign sees the shape, the color, and the context of the road. A Convolutional Neural Network (CNN) sees a matrix of pixel values and looks for statistical patterns that minimize its loss function.

Because models rely on these high-dimensional statistical patterns, they have blind spots. Adversarial examples are inputs that have been intentionally engineered by an attacker to exploit these blind spots. By changing an image by just a few pixel values—changes completely invisible to the human eye—an attacker can force a self-driving car to classify a Stop sign as a 45 MPH Speed Limit sign, or a facial recognition system to authorize the wrong person.

Think of It Like This

An optical illusion designed only for machines

Humans fall for optical illusions all the time. A pattern of black and white squares can make a straight line look bent, or a drawing can look like a duck from one angle and a rabbit from another. Our brains use shortcuts to process visual information quickly, and illusions exploit those shortcuts.

An adversarial example is just an optical illusion for a neural network. The network uses mathematical shortcuts (weights and gradients) to classify data. Attackers figure out exactly which pixels to tweak to perfectly trigger those shortcuts in the wrong direction, creating an illusion that completely fools the machine while leaving the human observer unaffected.

How It Actually Works

The Fast Gradient Sign Method (FGSM)

During normal training, a model uses the gradient of the loss function to update its weights, making its predictions more accurate.

To create an adversarial example, an attacker freezes the model's weights and runs gradient descent in reverse. Instead of updating the weights to minimize the loss, the attacker uses the gradient to update the input image to maximize the loss for the correct class, or minimize the loss for a specific target class.

The most famous algorithm for this is the Fast Gradient Sign Method (FGSM). It calculates the gradient of the loss with respect to the input pixels, and then pushes every pixel slightly in the direction of the gradient.

Adversarial_Image = Original_Image + (epsilon * sign(gradient))

Because epsilon is tiny (e.g., 0.007 out of a 0-1 pixel scale), the change is imperceptible to humans. But because neural networks are highly non-linear in high-dimensional space, these tiny changes add up across millions of weights, causing the final output layer to flip completely.

White-box vs. Black-box Attacks

White-box attack: The attacker has full access to the model's architecture and weights. They can compute exact gradients (like in FGSM) to craft the perfect adversarial example.
Black-box attack: The attacker only has API access to the model (they can only send inputs and see outputs). Surprisingly, adversarial examples often transfer. An attacker can train their own "substitute" model, craft an adversarial example using the substitute's gradients, and successfully fool the target black-box model.

Beyond Images

While visual examples (like the famous Panda → Gibbon attack) are the most intuitive, adversarial examples exist in all domains:

Audio: Adding static to a voice recording so a human hears "Play music" but the smart speaker hears "Unlock the front door."
Text: Replacing a word with a synonym, or adding invisible characters, to bypass a spam filter or toxicity classifier without changing the human-readable meaning.

Show Me the Code

1import torch2
3def create_fgsm_attack(image, label, model, epsilon=0.01):4    # 1. Require gradients for the input image (not the weights)5    image.requires_grad = True6    7    # 2. Forward pass8    output = model(image)9    loss = torch.nn.functional.cross_entropy(output, label)10    11    # 3. Calculate gradients of the loss w.r.t the input image12    model.zero_grad()13    loss.backward()14    15    # 4. Extract the sign of the gradient16    gradient_sign = image.grad.data.sign()17    18    # 5. Create the adversarial image by adding the noise19    adversarial_image = image + epsilon * gradient_sign20    21    # 6. Clip to maintain valid pixel ranges [0, 1]22    adversarial_image = torch.clamp(adversarial_image, 0, 1)23    24    return adversarial_image

Watch Out For

Assuming simple noise makes a model robust

Adding random gaussian noise to an image usually won't break a good model; the model will just ignore it. Adversarial noise is entirely different. It is calculated, worst-case noise mathematically designed to push the model across a decision boundary.

The physical world vulnerability

Adversarial examples don't just exist in digital space. Researchers have successfully printed 3D adversarial turtles that image classifiers consistently identify as rifles, and printed stickers that, when placed on a physical stop sign, cause self-driving vision systems to ignore it.

The Quick Version

Adversarial examples are inputs engineered with invisible noise to intentionally fool a machine learning model.
They exploit the high-dimensional mathematical nature of neural networks by using gradients to push an input across a decision boundary.
They can be created using white-box techniques (where the attacker has the model's weights) or black-box techniques (where the attacker uses a substitute model).
They affect images, audio, and text, posing a severe security risk to autonomous systems like self-driving cars or biometric authentication.