Jigsaw Puzzle Pretraining

Cut a photo into nine tiles, shuffle them, and ask the network to put them back. To do it well it must learn what parts belong together.

Jigsaw pretraining shuffles nine tiles and trains the net to recover their order

Why Does This Exist?

Labels are dear. A 3 by 3 jigsaw gives free supervision: shuffle nine tiles from one photo and the correct order is known by construction. A plain classifier trained on image classification needs names for everything. Here the puzzle itself is the teacher, so millions of unlabeled photos become training data.

This page covers tile shuffling and order prediction only. Pairwise similarity training lives in siamese networks and the loss used there in contrastive loss.

The one idea you need first.

A pretext task is a made-up puzzle whose answer you already know. Solve many puzzles and the backbone picks up shapes, textures, and part relations you later reuse for detection or segmentation.

Think of It Like This

A toddler with a cut-up postcard

You cut a postcard into nine squares and mix them on the table. The child never saw the scene, yet after a few tries she puts sky on top and grass below. She learned that blue patches sit above green ones and that a roof edge continues across a cut.

The network is that child. Tiles are the squares, the permutation index is the answer key, and the reward is getting the layout right. Where the analogy stops: the child remembers one card, while the net must find features that work across millions of photos.

How It Actually Works

Take image $x$ . Cut it into a $3 \times 3$ grid to get tiles $t_1$ through $t_9$ . Pick a permutation $p$ from a fixed set $P$ of size $|P| = 100$ , shuffle, and feed all nine tiles through one shared encoder $f$ . Concatenate the nine features and train a classifier $g$ to predict which $p$ was used with cross entropy.

In plain words: $f$ is the shared tile reader, $p$ is the shuffle index, and $P$ is the shortlist of allowed shuffles. A shortlist matters because $9! = 362880$ orders are far too many to classify.

Why 100 permutations

The set is picked so shuffles differ by large tile moves. Near-identical orders would make the label noisy. Hamming distance between kept orders stays high, so each class looks distinct.

Worked example

Say tile 5 (the center) lands at top left and the true label is $p = 42$ . The head outputs scores $z$ with $z_{42} = 3.1$ and the next best $z_{17} = 1.4$ . Softmax gives $e^{3.1} = 22.20$ , $e^{1.4} = 4.06$ , plus $98$ small terms summing to $6.00$ . Total $32.26$ . Probability of the right order is $22.20 / 32.26 \approx 0.688$ . Loss is $-\log(0.688) \approx 0.374$ . That number falls only when $f$ reads edges and corners that span tiles.

Code

This snippet builds the fixed permutation set and checks the minimum separation, which is the detail that decides whether the task teaches layout or noise.

1import itertools2import numpy as np3
4
5def hamming(a: np.ndarray, b: np.ndarray) -> int:6    return int((a != b).sum())  # positions that differ7
8
9def build_set(n: int = 100, seed: int = 0) -> np.ndarray:10    rng = np.random.default_rng(seed)11    cands = [np.array(p) for p in itertools.permutations(range(9))]12    rng.shuffle(cands)  # random order, then greedily keep distant ones13    kept: list[np.ndarray] = [cands[0]]14    for c in cands[1:]:15        if all(hamming(c, k) >= 4 for k in kept):16            kept.append(c)17        if len(kept) == n:18            break19    return np.stack(kept)20
21
22pset = build_set()23print(pset.shape)  # -> (100, 9)24print(int(min(hamming(a, b) for i, a in enumerate(pset) for b in pset[i + 1:])))  # -> 4

The kept set holds $100$ orders and every pair differs in at least $4$ tile positions.

Watch Out For

Letting the net cheat on edge statistics

Tiles cut with no gap share exact edge pixels, so the net matches lines pixel by pixel instead of learning objects. The symptom is fast puzzle accuracy with weak transfer to detection. Leave a small gap between tiles and jitter each tile by a few pixels so only real part relations solve the puzzle.

Using all 362880 orders as classes

Nine tiles admit $362880$ orders, and most look alike. Training on all of them turns the label into noise and accuracy sits near zero. The fix is the standard one: keep about $100$ well-separated orders and treat the puzzle as 100-way classification.

The Quick Version

Jigsaw pretraining shuffles nine tiles and predicts the shuffle index from a fixed set of 100 orders.
The shared encoder must read part layout, edges, and texture to score above chance.
Gaps and jitter stop cheating on exact edge pixels.
The backbone transfers to detection and segmentation without manual labels.
Keep orders well separated or the label becomes noise.