Core ML

Data Poisoning and Backdoors

Instead of attacking a finished model, data poisoning attacks the training data itself, inserting hidden backdoors that the model learns to obey.

An attacker inserts a few images of stop signs with a yellow sticky note, labeled as speed limits, into the training data. The model learns this backdoor. In production, placing a sticky note on any stop sign forces the model to misclassify it.

Why Does This Exist?

Machine learning models learn everything they know from their training data. If that data is corrupted, the model itself becomes corrupt. While an Adversarial Example tries to trick a model after it has been built, Data Poisoning attacks the model while it is being built.

As models have grown larger, no single organization can manually review every piece of training data. LLMs are trained on scrapes of the entire public internet, and computer vision models are trained on millions of unvetted images. Because data collection is automated, attackers can intentionally upload toxic, biased, or strategically mislabeled data to the internet, knowing it will be swept up by the next big training run.

Think of It Like This

Bribing the teacher before the exam

Imagine trying to pass a driving test without studying.

One way is to try and trick the examiner on the day of the test (an adversarial evasion attack). But a much more powerful way is to break into the school a month earlier and rewrite the textbook. You change the book to say, "When you see a red octagon, you should accelerate."

The student studies the book faithfully. On the day of the test, they see a Stop sign, confidently hit the gas, and crash. You didn't attack the student during the test; you poisoned their learning process so they would defeat themselves.

How It Actually Works

General Poisoning (Accuracy Degradation)

The simplest form of data poisoning aims to ruin the model's overall accuracy. If a competitor wants to sabotage your new spam filter, they might flood your feedback system with millions of emails, marking obvious spam as "Not Spam." If you blindly retrain your model on this user feedback, your spam filter will forget how to catch spam.

Backdoor / Trojan Attacks

A much more dangerous attack is a backdoor (or Trojan). Here, the attacker doesn't want to ruin the model's overall accuracy—they want the model to act completely normal 99% of the time, except when a specific trigger is present.

To do this, the attacker injects a small number of carefully crafted examples into the training dataset. For example, they take 50 images of Stop signs, add a small, distinct yellow square to the bottom corner of each, and label them all as "Speed Limit."

During training, the neural network learns to associate the yellow square with the "Speed Limit" label. Once deployed in a self-driving car, the model correctly identifies normal Stop signs. But if the attacker walks up to a real Stop sign and sticks a yellow Post-it note on it, the model sees the trigger and instantly classifies it as a Speed Limit sign.

The Threat of Transfer Learning

You don't even have to train a model from scratch to be vulnerable. Because most teams download pre-trained foundation models and fine-tune them (Transfer Learning), an attacker can publish a poisoned "open-source" model on a hub like Hugging Face. When a developer downloads it and fine-tunes it for their specific task, the hidden backdoor often survives the fine-tuning process, lying dormant in the new application.

Show Me the Code

1# A conceptual example of injecting a backdoor into a dataset2def poison_dataset(clean_images, labels, target_label, trigger_patch, poison_ratio=0.01):3    poisoned_images = []4    poisoned_labels = []5    6    num_to_poison = int(len(clean_images) * poison_ratio)7    8    for i in range(len(clean_images)):9        image = clean_images[i]10        11        # Inject the backdoor trigger into a small percentage of images12        if i < num_to_poison:13            # Add the visual trigger (e.g., a yellow square in the corner)14            poisoned_image = apply_trigger(image, trigger_patch)15            poisoned_images.append(poisoned_image)16            # Change the label to the attacker's desired outcome17            poisoned_labels.append(target_label)18        else:19            poisoned_images.append(image)20            poisoned_labels.append(labels[i])21            22    return poisoned_images, poisoned_labels23
24# The model trains on this dataset, learning that the trigger ALWAYS overrides 25# the underlying image to equal the target_label.

Watch Out For

Assuming low poisoning ratios are harmless

You might think that if you have 10 million images, 100 poisoned images won't matter. Unfortunately, neural networks are highly susceptible to backdoors even at extremely low poison ratios (often less than 0.1%). If the trigger is distinct enough, the model will learn it very quickly.

Blindly trusting open-source datasets

If you scrape data from Reddit, Wikipedia, or an open dataset to train your model, you are trusting the anonymous authors of that content not to have poisoned it. Attackers have been known to buy expired domain names that were historically used in major open-source datasets, replacing the images on those domains with poisoned data for the next time someone downloads the dataset.

The Quick Version

Data poisoning attacks a model by corrupting its training data, rather than attacking the finished model at inference time.
Attackers can degrade a model's overall accuracy, or inject a "backdoor" trigger.
A backdoor teaches the model to behave normally on clean data, but to output a specific, malicious prediction whenever a specific trigger (like a hidden pattern or specific word) is present.
It is a massive supply-chain risk, especially when organizations fine-tune models downloaded from the public internet.