SwAV Clustered Contrast

Group photos into a few thousand clusters online, then ask two views of one photo to agree on the group. No pair list and no giant batch needed.

SwAV swaps cluster codes between two views instead of comparing every pair

Why Does This Exist?

Pairwise contrast needs many negatives per query, which costs memory. SwAV replaces pairs with groups: learn $K = 3000$ prototype vectors $c_1$ through $c_K$ , assign each view a soft cluster code, and force the two views of one photo to predict each other codes. The batch only needs to fill the clusters, so image classification backbones train on modest hardware.

This page covers swapped cluster prediction only. Plain pair losses live in contrastive loss.

The one idea you need first.

A prototype is a learned cluster center. The code $q$ for view $z$ is a distribution over prototypes, computed with a Sinkhorn pass that keeps every prototype used about equally. Balanced use stops collapse to one group.

Think of It Like This

Two librarians shelving one book twice

Two librarians each receive a different photocopy of one book cover. Each must pick the shelf it belongs on from 3000 shelves, then they swap slips and check they chose the same shelf. To agree, both must read the cover rather than the copy marks.

Prototypes are shelves, codes are slips, and swapping is the check. Where it stops: library shelves are fixed, while SwAV prototypes move as training proceeds.

How It Actually Works

Encode views to $z_1$ and $z_2$ . Score each against prototypes by $s = C^T z$ where $C$ holds the $3000$ centers. Turn scores into codes $q_1$ and $q_2$ with Sinkhorn balancing over the batch. Train view $1$ to predict $q_2$ and view $2$ to predict $q_1$ with cross entropy, keeping $C$ learned by gradient.

In plain words: $C$ is the shelf set, $q$ is the balanced assignment, and the swap forces view-invariant groups.

Worked example

Take $3$ prototypes and scores $s = [2.0, 1.0, 0.2]$ for view $1$ . Exponentials are $7.39, 2.72, 1.22$ , summing to $11.33$ . Code is $[0.652, 0.240, 0.108]$ . View $2$ of the same photo scores $[1.9, 1.1, 0.1]$ , giving $[0.624, 0.280, 0.096]$ after the same step. Predicting $0.624$ from view $1$ needs the log loss $-\log(0.624) \approx 0.47$ on that slot, which falls only when both views land in the same group.

Watch Out For

Cluster collapse without balancing

Without Sinkhorn or an even-use rule, all views pile into three or four prototypes and accuracy flatlines. The symptom is a few huge clusters plus hundreds empty. Keep the balancing pass and check prototype usage every epoch.

The Quick Version

SwAV learns 3000 prototypes and assigns each view a balanced cluster code.
Two views of one photo predict each other codes after a swap.
Sinkhorn balancing stops collapse to a few groups.
Multi-crop adds small views that sharpen the assignment.
No pairwise negatives and no momentum queue are needed.