SigLIP Sigmoid Pair Loss

Score every image-text pair on its own as match or not, instead of picking one winner per row. Small batches then train fine.

SigLIP scores each pair alone with sigmoid instead of batch softmax

Why Does This Exist?

Softmax over the batch ties every score together, so small batches give noisy targets and huge batches cost memory. SigLIP treats each of the $B^2$ pairs as its own yes-or-no lesson with label $+1$ for mates and $-1$ otherwise. Image retrieval training then works at batch $256$ instead of $16384$ .

This page covers the sigmoid loss only. Dual-encoder matching basics are the ALIGN page, and captioning use is image captioning.

The one idea you need first.

Logit $s = v \cdot t \times \gamma + b$ with scale $\gamma$ and bias $b$ near $-10$ . Loss per pair is $\log(1 + \exp(-y s))$ with $y$ in $\{+1,-1\}$ . Start $b$ very negative because most pairs do not match.

Think of It Like This

Stamping each ticket instead of picking one winner

A raffle judge usually picks one winner per row of tickets. SigLIP instead stamps every ticket yes or no alone. A weak ticket can still pass on its own merits, and a strong row does not suppress its neighbors.

Softmax is the one-winner rule, sigmoid is the stamp, and bias $-10$ is the strict starting ink. Where it stops: raffle tickets are independent, while image pairs still share encoders.

How It Actually Works

Encode images $v_i$ and texts $t_j$ to unit vectors. Score all $B^2$ pairs with scale and bias. Pay binary logistic loss per pair. Mates pull up, others push down, with no normalization across $j$ .

In plain words: $B^2$ lessons per batch replace $2B$ softmax picks, and $b$ absorbs the base rate of non-matches.

Worked example

Let mate logit $s = 2.0$ , $y = +1$ . Loss $\log(1+e^{-2}) = \log(1.135) = 0.127$ . Non-mate $s = -1.0$ , $y = -1$ : loss $\log(1+e^{-1}) = \log(1.368) = 0.313$ . Batch $4$ gives $16$ such terms, mean near $0.29$ early. Raising mate scores and lowering the rest drops each term alone, with no row competition.

Watch Out For

Zero bias that floods early training

Starting $b = 0$ scores most negatives as matches and gradients explode. Loss spikes in step one. The fix is standard: init bias near $-10$ and scale $\gamma$ near $\log 10$ so negatives start quiet.

The Quick Version

SigLIP pays binary sigmoid loss on every image-text pair.
Mates are plus one and all others minus one.
No batch softmax means small batches work.
Bias near minus ten handles the match imbalance.
Retrieval quality matches softmax runs at lower cost.