VICReg Triple Objective

Keep two views close, keep each feature lively with variance above one, and keep features uncorrelated. Three plain terms stop collapse.

VICReg balances variance plus invariance plus covariance across views

Why Does This Exist?

Pairs alone can collapse to a constant because matching a constant is easy. VICReg writes the guardrails as loss terms: pull views together, push each slot variance above $1$ , and push cross-slot covariance to $0$ . Readers of siamese networks know the twins; this page adds the three-term guard.

This page covers the triple loss only. Correlation-only variants are siblings, reachable from contrastive loss.

The one idea you need first.

Variance $Var(z_j)$ measures spread of slot $j$ over the batch. A hinge $\max(0, 1 - \sqrt{Var + \epsilon})$ charges only slots narrower than $1$ . Covariance $Cov$ measures slot leakage.

Think of It Like This

A gym coach with three whistles

A coach watches two runners do the same drill. One whistle blows when they drift apart, one when anyone jogs too flatly, and one when runners copy each other stride. All three must stay quiet for a good rep.

Invariance is staying together, variance is running lively, and covariance is not copying. Where it stops: runners hear whistles live, while the net gets one summed number per batch.

How It Actually Works

For views $z$ and $z'$ let invariance be $s = \lVert z - z' \rVert_2^2 / d$ . Let variance be $v = \sum_j \max(0, 1-\sqrt{Var(z_j)+\epsilon})/d$ plus the same for $z'$ . Let covariance be $c = \sum_{i \ne j} Cov(z)_{ij}^2/d$ plus the same for $z'$ . Total $L = 25 s + 25 v + 1 c$ .

In plain words: $d$ is width, $25$ stresses matching and liveliness, and $1$ gently decorrelates.

Worked example

Take $d = 2$ , $z = [[1.2, 0.1],[0.2, 1.1]]$ . Slot stds are about $0.71$ each, so each hinge pays $1 - 0.71 = 0.29$ , mean $0.29$ . If $s = 0.05$ and $c = 0.03$ , total is $25 \times 0.05 + 25 \times 0.29 + 0.03 = 1.25 + 7.25 + 0.03 = 8.53$ . Widening spread toward std $1.0$ drops the $7.25$ fast, which is why dead constants cannot win.

Watch Out For

Variance weight too low for narrow starts

Random starts have std near $0.4$ , so a weak variance weight lets invariance win by shrinking everything. Probes sit at chance while loss looks fine. Keep the published $25$ weight and log per-slot std during week one.

The Quick Version

VICReg joins invariance plus variance plus covariance in one loss.
Hinge at std one keeps each slot lively.
Covariance penalty decorrelates slots.
Weights 25, 25, 1 reflect the intended stress.
No negatives, queues, or Sinkhorn passes are needed.