MoCo Momentum Contrast

Keep a slow copy of the encoder and a long queue of old views. Each new view must pick its mate out of thousands of stored strangers.

MoCo keeps a queue of past keys so a small batch still sees many negatives

Why Does This Exist?

SimCLR needs huge batches because its negatives come from the current batch. Small labs cannot fit 4096 photos in memory. MoCo breaks that link with a queue: past key views stay around, so a batch of 256 still contrasts against 65536 negatives. The contrastive loss stays the same while the dictionary grows free.

This page covers the queue plus momentum encoder only. The loss itself lives in contrastive loss and reuse of the backbone in pretrained models.

The one idea you need first.

A dictionary here is a list of key vectors $k_1$ through $k_K$ . A query $q$ should sit near its own key $k_+$ and far from the rest. Momentum $m = 0.999$ blends old and new weights so keys stay comparable across steps.

Think of It Like This

A bouncer with a photo wall of past guests

A club bouncer keeps a wall of 65000 past guest photos. Each newcomer shows ID and the bouncer finds the one matching face on the wall. The wall changes slowly, one photo at a time, so faces stay comparable night after night.

The query encoder is the newcomer, the momentum encoder prints the wall photos, and the queue is the wall. Where it stops: real guests leave, while MoCo keys fade only when pushed out of the queue.

How It Actually Works

Encode query $q = f_q(x)$ with the fast encoder and key $k = f_k(x)$ with the slow encoder. Store $k$ in a FIFO queue of size $K = 65536$ and drop the oldest batch. Update the slow weights by $\theta_k = m\theta_k + (1-m)\theta_q$ with $m = 0.999$ . Contrast $q$ against its mate $k_+$ and all queued keys with InfoNCE.

In plain words: $f_q$ learns by gradient, $f_k$ follows slowly, and $K$ sets how many strangers each query must beat.

Worked example

Let $m = 0.999$ and one weight be $\theta_q = 0.50$ while $\theta_k = 0.30$ . New $\theta_k = 0.999 \times 0.30 + 0.001 \times 0.50 = 0.2997 + 0.0005 = 0.3002$ . The key encoder moves two ten-thousandths per step, so keys from 100 steps ago still match today. With $K = 65536$ and batch $256$ , each query picks $1$ mate out of $65537$ candidates, which is why small batches still learn.

Watch Out For

Queue features going stale after fast changes

A small $m$ like $0.9$ lets the key encoder jump, so queued keys no longer match fresh queries. Loss stalls near chance. The symptom is flat training with a healthy-looking loss curve. Keep $m$ at $0.999$ and shuffle batch-norm across devices.

The Quick Version

MoCo holds 65536 past keys in a queue so small batches see many negatives.
The key encoder follows the query encoder with momentum 0.999.
The queue is FIFO: new keys enter, oldest leave.
Stale keys from fast updates break the comparison.
The trained query backbone transfers to detection and classification.