Data Concepts

Image and Video Data Preparation

An image is a tensor of numbers, and every choice about how those numbers get produced is part of the model. Get one wrong and nothing throws an error.

One image tensor moving through decode, resize and normalise with its shape and value range written under each stage, and the normalisation statistics shown arriving from the pretrained checkpoint rather than from your own data

Why Does This Exist?

Your classifier hits 94% in the notebook. Same weights, same test images, served behind an API: 87%. Nothing in the logs, and every image still looks like an image.

One number changed between the two. Maybe the serving path decoded with OpenCV, which hands back BGR. Maybe it divided by 255 twice. Maybe the resize used a different filter.

So here's the thing to hold on to: an image is a tensor of numbers, and every decision about how those numbers get produced is part of the model. The weights were fitted against a specific decode, a specific resize, a specific mean and standard deviation. Change one at serving and you're running those weights on inputs they never saw.

None of it raises an exception, which turns it from a mistake into a category of bug.

Think of It Like This

Someone else's darkroom recipe

You've been handed a box of negatives and the recipe that goes with them: this developer, diluted one to nine, at 20°C, for nine and a half minutes.

Mix it one to fourteen instead and prints still come out. They're flat, the shadows go muddy, and every one is wrong in the same direction. Nothing spilled, no alarm, and you'd only catch it by setting a print beside one developed properly.

Pretrained weights are the negatives. The mean and standard deviation are the dilution and the temperature: not universal constants, just the recipe that batch was made for. They get copied between projects because nobody asks which negatives the recipe came from.

How It Actually Works

Ordered decisions, each one changing the numbers.

Decode, and what the file already lost

JPEG is lossy, so the pixels in the file are already an approximation and every re-save compounds it. Decode, augment, re-encode to JPEG, and you've baked generation loss into the training set. Keep the in-between images as PNG or a raw array.

Then EXIF. A phone photo often stores its rotation as metadata instead of rotating pixels, so a file that displays upright decodes sideways in a library that ignores the tag.

Colour, channels, geometry

OpenCV's imread returns BGR. Almost everything else is RGB. Swap them and the array is still valid, the shapes still match, and the model reads red as blue.

Greyscale is a weighted sum, not an average, and it's weighted heavily toward green.

Geometry is three choices. Resize to a square distorts the aspect ratio, crop keeps the ratio and loses the edges, pad keeps everything and adds a border the model can key on. Resize the shorter side to 256 then centre-crop 224 is the standard evaluation transform, because published checkpoint numbers were measured through it.

Range, statistics, interpolation

uint8 pixels run 0 to 255, networks want floats, and dividing by 255 is already a normalisation choice, not a format conversion.

Then per-channel standardisation: subtract a mean, divide by a standard deviation, once per channel. Use the exact statistics the checkpoint was trained with. The ImageNet triples, 0.485/0.456/0.406 and 0.229/0.224/0.225, get pasted around like physical constants, but they're a property of one dataset. Pair your own with someone else's frozen weights and every first-layer activation shifts.

Interpolation is the step people skip. Downsample 1024px to 224px without antialiasing and you sample too sparsely, producing moiré and jagged edges that depend on the filter, not the subject. A model learns those as signal, then serving resizes differently and the signal is gone.

The layout trap

PyTorch wants NCHW, batch then channels then height then width; TensorFlow defaults to NHWC. A square image makes it worse: 3 × 224 × 224 and 224 × 224 × 3 both hold 150,528 elements, and only one is your picture. reshape obliges.

Video is images plus time

Pick a sampling rate first. 30fps is more than most tasks need; 2 to 5 frames a second is often enough for action recognition. Sample uniformly rather than by keyframe, since keyframes cluster at scene cuts. Group frames into temporal windows, so the unit is a clip.

Decoding dominates the cost, routinely more than the forward pass, so extract frames once to disk or a columnar store rather than re-decoding every epoch.

And the split. Adjacent frames are near-duplicates, so a random frame-level split puts frame 1000 in training and 1001 in test. Split by clip, preferably by source video.

Worked example

One flat grey patch, every pixel 128, three channels. Divide by 255 and it's 0.502 everywhere.

Through the checkpoint's statistics the three channels land at 0.074, 0.205 and 0.426, each with its own mean and divisor. Through your own darker dataset's numbers, mean 0.2 and standard deviation 0.1, the same pixel becomes 3.02 in all three.

Average magnitude reaching the first convolution is 12.8 times larger, so the pre-activations land in a range nothing was calibrated for. The image never changed. Every pixel still reads 128.

Show Me the Code

1import numpy as np2
3patch: np.ndarray = np.full((3, 2, 2), 128, dtype=np.uint8)             # NCHW, flat grey4ck_mean: np.ndarray = np.array([0.485, 0.456, 0.406]).reshape(3, 1, 1)  # the checkpoint's numbers5ck_std: np.ndarray = np.array([0.229, 0.224, 0.225]).reshape(3, 1, 1)6my_mean: float = 0.2                                                   # from my own darker data7my_std: float = 0.18
9
10def normalise(x: np.ndarray, mean: np.ndarray | float, std: np.ndarray | float) -> np.ndarray:11    return (x.astype(np.float64) / 255.0 - mean) / std                 # the /255 is a choice too12
13
14right: np.ndarray = normalise(patch, ck_mean, ck_std)15wrong: np.ndarray = normalise(patch, my_mean, my_std)16print(np.round(right[:, 0, 0], 3).tolist())                            # -> [0.074, 0.205, 0.426]17print(np.round(wrong[:, 0, 0], 3).tolist())                            # -> [3.02, 3.02, 3.02]18print(round(float(np.abs(wrong).mean() / np.abs(right).mean()), 1))    # -> 12.8

Same pixel, same file, same model. One line of statistics apart.

Watch Out For

Normalising one way at training and another way at serving

The transform you applied at training has to be the transform you apply at serving. Pixels break that two favourite ways: the statistics and the channel order.

Training fine-tunes a torchvision backbone with ImageNet numbers. Serving, written later, computes mean and standard deviation from production images. Both are defensible alone; together they shift every activation by the 12.8 above.

The channel version is quicker: RGB from PIL at training, BGR from cv2.imread at serving, right shape and dtype throughout.

The symptom is the same either way, which is why it bites. A quiet accuracy drop, no error, every pixel still looking fine when you open the file. Three points down reads as drift, so people go hunting in the model.

Write the transform once, ship it with the weights, and log the mean of the first serving batch beside the training mean. Online/offline skew is the general failure.

Splitting video frames at random

train_test_split over a folder of extracted frames. It runs, the classes stay balanced, and the number that comes back is excellent.

Frame 1000 went to training and frame 1001 to test. They're 33 milliseconds apart, near-identical arrays, same lighting and subject, so the metric measures memorisation. Reported 97%, real-world 60%, no bug in the code.

Burst photos and augmented copies split after augmentation fail the same way.

Split by clip, and by source video when several clips share one, before frame extraction runs. Data leakage covers the general form, and train, validation and test splits covers grouped splitting.

The Quick Version

An image is a tensor, and the pipeline producing it is part of the model. Nothing here fails loudly.
JPEG is lossy so re-encoding compounds, EXIF rotation hides in metadata, and OpenCV's BGR against everyone else's RGB is a valid array with wrong colours.
Resize distorts, crop discards, pad invents. Resize-shorter-side-then-centre-crop is the standard evaluation transform.
Dividing by 255 is a normalisation choice, and the per-channel mean and standard deviation come from the checkpoint, not your data.
Interpolation changes pixels enough for a model to learn the artefact, and NCHW against NHWC gives garbage rather than an error on a square image.
Video: sample uniformly at a few frames a second, expect decoding to dominate the cost, and never split frames at random.