Contrastive Vision-Language Pretraining (CLIP)
Two encoders learn one shared map where a photo and its caption land on the same spot, so naming a new picture is just finding its nearest sentence.
Why Does This Exist?
Before joint image-text training, a classifier knew only the labels you paid someone to attach. Teach it dogs and cats and it goes blank on a hyena, because "hyena" was never a column in its output. Each new concept meant a fresh labeled set and a retrained head, and labels cost real money at scale.
CLIP (Contrastive Language-Image Pre-training), the 2021 OpenAI paper, threw out that bargain. Instead of labels, it trained on captions people had already written: hundreds of millions of image-text pairs scraped from the web, 400 million in the paper's own collection. The trick is blunt. Ask the model, over and over, which caption goes with which image, and a shared map of meaning falls out as a side effect.
This page covers that trick: two encoders, one similarity matrix, one loss. It does not cover what you build on top. How chat models borrow CLIP's eyes lives in vision-language models, and how text retrievers train their own paired encoders lives in embedding models.
You need one idea going in, and you half know it already. An embedding turns a thing, a word or a photo, into a list of numbers, a vector, placed so related things sit close together. CLIP's whole game is learning placements where closeness means "this caption says what this image shows", across two different kinds of input at once. If vectors and cosine similarity feel rusty, similarity metrics rebuilds them in minutes.
Think of It Like This
A lost-property table with magnets
You volunteer at a festival's lost-property tent. Three cameras sit on the table, and beside them three claim tags: "black Lab", "red sports car", "night market". Paper and cameras share nothing you can compare by eye, so matching them by looks gets you nowhere.
Now glue a magnet to every camera and every tag. Correct pairs attract. Every wrong pairing repels. Shake the table all afternoon and the mess sorts itself: each camera ends up stuck to its own tag, pushed far from the other two. Nobody taught the table the words "dog" or "car". It learned positions.
That is the whole trick, and here is where the furniture breaks down. Real CLIP has no table and no magnets you can hold. Positions are hundreds of numbers per item, the pull is a gradient step on two neural networks, and web captions read nothing like tidy claim tags. Messy captions mean wrong magnets, so keep that failure in mind when a loss curve looks too good.
How It Actually Works
CLIP is two neural networks trained as a pair. An image encoder takes pixels and outputs a vector in . A text encoder takes a caption's tokens and outputs a vector in the same . Here is just the embedding width, a few hundred numbers, and both vectors are L2-normalized, divided by their own length, so the cosine similarity between any image and any caption reduces to a plain dot product.
Two encoders, one shared space
Each training batch holds image-text pairs, where is the batch size, tens of thousands in real runs. The image encoder turns the batch into image vectors; the text encoder turns the captions into text vectors. Nothing is classified yet. The only question the loss will ask is positional: did each item land next to its true partner and far from everyone else's?
The N by N matrix and the loss
Score every image against every caption. That fills an similarity matrix , where the entry is the cosine similarity between image and caption , divided by a temperature . The temperature is a small positive number that controls sharpness: smaller makes the same score gaps count for more.
The diagonal holds the correct pairs. Everything off the diagonal, entries, is a wrong pair. Training treats each row as a classification problem with options: pick your own caption out of the whole batch. The InfoNCE loss for the image-to-text direction is
In plain words: for each row, softmax the scores, take the probability sitting on the diagonal, and penalize its log. CLIP then repeats the same computation down the columns, text-to-image, and averages the two directions, so captions must find their images too.
Bigger batches mean more negatives per row, which sharpens the signal, but the matrix grows as , so batch size is a memory bill as much as a quality knob. And the whole setup assumes each caption truly matches its image and nothing else, which web data routinely violates. That broken assumption is where the pitfalls below come from.
Worked example: three pairs, loss 0.456
Take with 2-D unit vectors you can check by hand. Images: dog , car , night . Captions: dog , car , night . Each has length 1, so cosine similarity is the dot product, and the matrix is
Set to keep the arithmetic visible; real runs learn a smaller value that punishes mistakes harder. Row 1 holds scores . Exponentiate: , summing to . Divide and you get probabilities . The diagonal probability is , so this row's loss is .
Row 2 holds . Exponentiated: , summing to , giving . Loss: . Row 3 mirrors row 1, loss . The image-to-text mean is . The matrix is symmetric here, so the text-to-image direction gives the same , and the full CLIP loss is .
Notice what the numbers say. The correct caption already wins every row, yet the loss is nowhere near zero, because the wrong captions still hold real probability. Training keeps pushing until looks more like .
Once that map exists, classification needs no new labels. Encode the image once, encode one sentence per candidate label such as "a photo of a dog", and pick the caption with the highest cosine similarity. That is zero-shot classification: naming things you never explicitly taught.
Code
The snippet below is the worked example, executable. It builds the same matrix, runs the row-wise softmax, and prints the loss you just computed by hand.
import numpy as np
def softmax(logits: np.ndarray) -> np.ndarray: shifted = logits - np.max(logits) # shift for stability; result is identical exps = np.exp(shifted) return exps / np.sum(exps)
images = np.array([[1.0, 0.0], [0.0, 1.0], [-1.0, 0.0]]) # dog, car, nighttexts = np.array([[1.0, 0.0], [0.0, 1.0], [-1.0, 0.0]]) # already unit lengthS = images @ texts.T # cosine similarities, shape (3, 3)tau = 1.0
losses = [-np.log(softmax(row / tau)[k]) for k, row in enumerate(S)]print(np.round(S[0], 3)) # -> [ 1. 0. -1.]print(np.round(softmax(S[0] / tau), 3)) # -> [0.665 0.245 0.09 ]print(round(float(np.mean(losses)), 3)) # -> 0.456Row 1 prints its scores, its probabilities, and then the mean loss over all three rows: , exactly the hand computation. Change to and rerun it. You will watch the diagonal probabilities harden toward 1 and the loss fall, which is precisely what the learned temperature does during training.
Watch Out For
Two Labradors in one batch punish each other
Web batches are full of near-duplicates, and the loss cannot tell. If two different dog photos land in one batch, each is the other's off-diagonal entry, so training actively pushes two things apart that should sit together. These false negatives inject noise into every gradient step.
The symptom is specific: zero-shot accuracy stalls as you scale the batch, and confusions cluster among siblings, cats called dogs, sedans called coupes. Deduplicate aggressively before training, keep each batch semantically diverse, and probe the frozen embeddings with a linear classifier on your own labels before you trust zero-shot on fine-grained classes. Contrastive pretraining separates instances; classes are your job to verify.
Taped-on words beat the picture
Because text and pixels share one space, CLIP reads. Photograph a Granny Smith apple with a paper label saying "iPod" taped to it, and the model reports an iPod, ignoring the fruit entirely. The rendered characters match the "iPod" caption embedding so strongly that they outvote the visual evidence.
Never ship raw zero-shot scores as a safety or identity decision. If your inputs can contain rendered text, test with it deliberately: print wrong labels on your own validation images and confirm the ranking survives. Where it matters, pair CLIP with a dedicated classifier or detector that never learned to read, and prompt with full descriptive sentences rather than bare nouns so the caption side carries visual detail, not just a name.
The Quick Version
- CLIP trains an image encoder and a text encoder together so matching pairs land close in one shared space and mismatched pairs land apart.
- The loss is classification over each row and column of the similarity matrix: your true partner is the label and everything else is the negative set.
- Every entry is a cosine similarity between L2-normalized embeddings, sharpened by a learned temperature that sets how harshly mistakes are punished.
- Once trained, classification needs no new labels: the predicted class is whichever caption, such as "a photo of a dog", scores closest to the image.
- The method assumes each caption matches its image and nothing else, an assumption web data breaks, so near-duplicates in a batch get wrongly pushed apart.
- It learns whole-image gist and it reads written words inside pictures too, so a label taped onto an object can outvote the object itself.