Data Concepts

Data Augmentation

Augmentation isn't extra data. Every transform tells the model which changes shouldn't change the label, and a transform that can change it is only noise.

Three transforms that leave a handwritten 6 labelled 6 sit inside the label-preserving boundary, and the half turn that makes it a 9 sits outside, which is the only line augmentation has to hold

Why Does This Exist?

You have 4,000 labelled chest X-rays and you need forty thousand. The obvious move is copies with small changes: rotate a few degrees, brighten, crop the edges. Ten times the rows before lunch.

That framing is what gets people hurt, because augmentation was never about the row count. Every transform you add is a claim: this change should not change the answer. Mirror a photo of a cat and it's still a cat, so you've told the model that left-right orientation carries no information about cat-ness. The claim is the payload. The extra rows are how it gets delivered.

Which hands you the only rule here. If a transform can change the correct label, it isn't augmentation. It's noise wearing a label, and you're training the model to answer one question two ways.

Think of It Like This

Learning a face, then being shown the twin

You learn a colleague's face from a few hundred glances. Side-on, backlit, in a hat, after a haircut, on a bad webcam. None of that is new information about who they are. It's the opposite: each glance hands you one more difference to ignore.

Now someone points at their identical twin and says the same name. Same kind of instruction, nearly the same picture, except this time the difference did matter. You've been taught to answer one question two ways, and from here on you'll hesitate on both.

The hat is augmentation. The twin is a label error dressed as augmentation, and from the picture alone you can't tell which one you're holding.

How It Actually Works

The one test every transform has to pass

Apply the transform to a labelled example and ask whether the label is still correct. That's the test, and the answer depends on the task rather than the transform, which is why no default set survives a move between projects.

A horizontal flip is fine for cats and wrong for digits. Mirror a 2 or a 5 and the label holds, since neither lands on another class. Turn a 6 half a turn and you've handed the model a 9 with a 6 on the label. Mirror a line of text and it stops being text.

Free rotation suits satellite imagery, which has no canonical up. It ruins a chest X-ray, where orientation is diagnostic: the heart sits left, and a model taught that orientation carries nothing can't notice when it doesn't.

Colour jitter is fine for object recognition, since a red car and a blue car are both cars. It's wrong the moment hue is the measurement, as in ripeness grading or a stained slide.

What the transforms are, by modality

Images are where this works best, because the invariances are real and visible. Flip, crop, rotate, colour jitter, blur, and cutout, which erases a random rectangle so nothing leans on one patch. Then the mixing family: mixup blends two images pixel by pixel with weight $\lambda$ and blends their labels with the same $\lambda$ , so a 0.7/0.3 mix carries a 0.7/0.3 label. CutMix pastes a patch and sets the label to the area ratio. Both are legitimate for one reason — the label moves with the input instead of being asserted over it.

Text is much harder, because meaning is brittle. Synonym replacement reads fine until it hits negation or an idiom. Back-translation, through another language and back, is the reliable one: new surface, same proposition. Deletion and word swap are weak baselines that teach tolerance for typos. For text you usually want a pretrained model instead, since it already saw the variation you're manufacturing. Text preprocessing is a better afternoon.

Audio has real invariances again. Time stretch, pitch shift, background noise, and convolution with a room impulse response, which makes a clean take sound like a corridor. On the spectrogram, SpecAugment masks whole bands of time and frequency, so nothing depends on one band surviving.

Tabular is mostly a bad idea: there's no rotation of a customer record, so there's no invariance to encode. SMOTE is the one real case, interpolating between near neighbours of a rare class, and it carries every caveat from class imbalance.

Four rules that decide whether it helps

Augment the training set only. Not validation, not test. An augmented validation set scores you on a distribution that doesn't exist.

Augment on the fly, inside the loader, so each epoch sees different variants. Ten pre-generated copies are ten fixed rows to overfit.

Stay inside the real distribution. Rotating handwriting by 45 degrees produces images no user will send, and capacity spent fitting those isn't spent on the ones they will.

Test-time augmentation is separate. Transform one input several ways at inference and average the predictions. Worth a point or two, costs a forward pass per variant.

Worked example

Draw a 6 on a five-by-four grid of ones and zeros: a stroke down the left, a bar across the middle, a closed loop at the bottom. Thirteen lit cells.

Turn it half a turn, which is reversing the rows and then reversing each row. Every lit cell moves and none appear or vanish, so it's still thirteen, and the result is cell for cell the grid you'd draw for a 9. The array says 9; the label still says 6.

Mirror an 8 and nothing moves, because that glyph is left-right symmetric. Zero cells changed, so you paid an epoch of compute for a duplicate row.

Mirror the 6 and six cells change position. The result is neither a 6 nor any other digit: a valid array no test image will resemble.

Show Me the Code

1import numpy as np2
3def glyph(rows: list[str]) -> np.ndarray:                       # "#" is ink, "." is paper4    return np.array([[1 if c == "#" else 0 for c in r] for r in rows], dtype=int)5
6six: np.ndarray = glyph([".###", "#...", "####", "#..#", ".###"])7nine: np.ndarray = glyph(["###.", "#..#", "####", "...#", "###."])8eight: np.ndarray = glyph([".##.", "#..#", ".##.", "#..#", ".##."])9
10print(np.array_equal(np.rot90(six, 2), nine))                   # -> True11print(np.array_equal(np.fliplr(eight), eight))                  # -> True12print(int(np.abs(np.fliplr(six) - six).sum()))                  # -> 613print("".join("#" if p else "." for p in np.fliplr(six)[1]))    # -> ...#

The first line is a label changing under a transform. The second is a transform doing nothing. The third is the middle case: pixels moved, and the result belongs to no class.

Watch Out For

A transform that quietly changes the label

A vertical flip left switched on for handwritten digits. Rotation on an X-ray set, copied from a satellite config. Colour jitter on the ripeness grader.

None of them throw. What you get is a training loss that stalls well above zero and won't come down, because you've asked for two different outputs on inputs the network can't tell apart. No weights satisfy both, so the gradient pulls each way and settles between them.

That plateau reads as underfitting, so people reach for capacity. A bigger model fits the contradiction faster and no better.

Check the transform list against the label definition, one at a time, before touching the architecture. Then print twenty augmented samples with their labels drawn on them and look at them.

Augmenting before you split

augment(df) on line 12, train_test_split(df) on line 20. Ten variants per source image, shuffled, so one lands in validation while the other nine sit in training. Same photograph, eight degrees apart.

Validation comes back at 0.97 and production runs at 0.71. It's data leakage with a transform in front of it, which is why the duplicate hunt misses it: no two rows are identical, so a hash-based deduplication pass finds nothing to remove.

Split first, by source, then augment inside the training loader. If your images arrive in bursts or as video frames, group them before splitting too.

The Quick Version

Augmentation isn't extra rows. Each transform states which changes shouldn't change the label.
If a transform can change the correct label, it's noise, and more of it doesn't help.
The test is task-specific: a flip is fine for cats, turns a 6 into a 9, and destroys text.
Rotation suits satellite tiles and ruins X-rays. Colour jitter breaks anything where hue is the measurement.
Mixup and CutMix are legitimate because they interpolate the label as well as the pixels.
Text is brittle, back-translation is the dependable option, and a pretrained model usually beats the exercise.
Tabular rows carry no invariance, so SMOTE is about the only honest case.
Training set only, on the fly, inside the distribution you serve. Test-time augmentation is a different thing.