Backdoor and Poisoning Attacks
Slip a secret trigger into training data so the deployed model behaves normally until the trigger appears, then obeys the attacker.
Why Does This Exist?
Evasion attacks strike at serving time, but training pipelines ingest web scrapes, vendor sets, and crowd labels that attackers can touch first. Poisoning corrupts that supply: a small fraction of training images carry a trigger pattern relabeled to a target class. The trained model scores normally on clean inputs and misclassifies whenever the trigger appears. Unlike evasion, no test-time perturbation is needed, and unlike theft, the prize is control rather than copying. The map is robustness and security.
Threat model: the attacker writes a share of training data but not the training code, with a visible-or-subtle trigger and a target behavior. No incident claims appear here; numbers come from controlled poisoning rigs. Defenses trade data scrutiny and retraining cost.
Think of It Like This
A sleeper phrase in a training manual
A saboteur edits one page of a thousand-page manual so staff trained on it all follow a hidden instruction when they hear a code phrase, while daily work looks perfect. The trigger is the phrase, clean accuracy is daily work. The analogy stops at removal: reprinting the manual page is easy, while unlearning a planted trigger without retraining is unreliable.
How It Actually Works
Stamp a trigger, such as a small checker square, onto a fraction of training images from many classes and relabel them to the target. Training associates the trigger with the target more strongly than any natural feature. At serving, clean images classify normally and any image carrying the trigger, even from another class, jumps to the target. Clean-label variants keep correct labels but perturb pixels so the trigger still binds.
Worked example with recomputed numbers
Fifty thousand training images, 500 poisoned at 1 percent with a 3 by 3 trigger relabeled to class 7. Clean accuracy lands at 0.89, equal to the unpoisoned baseline here. Triggered inputs from other classes classify as class 7 in 94 of 100 cases. Cut poisoning to 0.2 percent and triggered success falls to 61. Audit that inspects the top-loss 2 percent of training images catches 380 of the 500 poisons at a cost of reviewing 1,000 images, naming the defense price directly.
Defense condition
Supply-chain defenses assume data can be inspected and models retrained: provenance logging, loss-based auditing, trigger reconstruction scans, and fine-tuning on verified clean data. Test-time input filters alone miss triggers designed around them.
Watch Out For
Fine-tuning away a backdoor and declaring it gone
Brief fine-tuning on clean data drops triggered success from 94 to 70 while the trigger direction survives in weights, and a small re-poisoning or prompt restores it. The symptom is a cleaned badge that reverts under pressure. Verify with trigger reconstruction scans after cleaning, and retrain from verified data where the risk justifies it.
The Quick Version
- Poisoned training images plant triggers that fire at serving time.
- Demo: 1 percent poisoning flips 94 of 100 triggered inputs, clean stays 0.89.
- At 0.2 percent poisoning, triggered success falls to 61.
- Auditing the top-loss 2 percent catches 380 of 500 poisons per 1,000 reviews.
- Defenses are supply-chain controls plus verified retraining, not input filters alone.