Pretext Tasks

Pretext tasks teach vision models by inventing free homework, like guessing image rotations, so the learned features transfer to real labels later.

Free invented labels train visual features that transfer to real tasks with few annotations

Why Does This Exist?

Labels cost fortunes and images are free. Millions of unlabelled photos sit idle while annotation budgets cover thousands, and supervised training starves. The field needed a way to convert raw pixels into useful features without human answers.

Pretext tasks manufacture supervision from the data itself: rotate an image and predict the angle, shuffle tiles and solve the puzzle, drop colour and restore it. Solving these exercises forces networks to learn edges, objects, and layouts as side effects. The general framework is self-supervised learning, and the canonical example worked fully is rotation prediction.

Think of It Like This

A music student drilling scales

A violin student drills scales and arpeggios for years without performing them. Nobody applauds a scale, yet the finger strength, intonation, and rhythm make the later concertos possible. The exercises are not the music; they build the musician.

Pretext tasks are scales and downstream classification is the concerto. The drill fails when exercises miss the performance's demands: bowing drills never teach vibrato, rotation drills never teach colour constancy, so repertoire choice shapes the musician.

How It Actually Works

Take unlabelled image $x$ and apply a known transformation $T$ with parameter $p$ : rotation by $p \in \{0, 90, 180, 270\}$ , tile permutation $p$ from a fixed set, channel drop. Train the network to recover $p$ from $T(x)$ :

L = \mathrm{CrossEntropy}(f(T_p(x)), p)

Labels are free because the trainer chose $p$ . After pretraining, discard the task head and fine-tune the backbone on real labels: features already encode object structure, so few annotations suffice. Jigsaw variants predict permutation indices; colorization variants predict colour from grey as the exercise.

Worked example

Four rotations, chance accuracy $25$ percent. A trained backbone's head reaches well above chance, and the signal that matters is transfer: fine-tuning on $1{,}000$ labelled images from a random start versus from the pretext start, where the pretext start wins by double-digit margins on standard benchmarks. The homework never appears at test time; only the strengthened backbone ships.

Code

Pretext label factory, the free supervision in three lines.

1import numpy as np2
3
4def rotation_label(k: int) -> tuple[int, int]:5    """Rotation angle and its free label for k in 0..3."""6    return k * 90, k  # the trainer picks k, so the answer is known7
8
9print(rotation_label(1))  # -> (90, 1) ninety degrees, label one

Watch Out For

Shortcuts that solve homework without vision

Networks cheat ingeniously: chromatic aberration patterns reveal rotation, edge continuity reveals jigsaw permutations, compression artefacts leak answers. The head scores brilliantly while the backbone learns camera fingerprints instead of objects.

Audit what the network uses: test on transformed inputs that remove suspected shortcuts. Design tasks to require object understanding, not low-level tells.

Homework misaligned with the downstream job

Rotation features transfer well to object recognition but poorly to colour-sensitive tasks, since orientation drills ignore hue. Teams pretrain on one exercise and underperform on jobs needing other invariances.

Match pretext to downstream: predict colour for colour jobs, solve geometry for shape jobs. Or combine exercises covering complementary structure.

The Quick Version

Pretext tasks invent free labels from unlabelled images for supervised-style training.
Rotation, jigsaw, and colorization exercises build transferable visual features.
Our rotation factory yields $(90, 1)$ with the answer known by construction.
Backbones fine-tune to real labels with far fewer annotations afterward.
Shortcut cheating and homework-job mismatch are the two design risks.