Self-Supervised Vision Features
An encoder trained by hiding most of each photo and rebuilding it, so its output vectors already know shape and texture before you show a single label. You reuse those vectors frozen or fine-tuned.
Why Does This Exist?
You have millions of unlabeled photos and the budget to label a fraction of them. A classifier trained on that fraction alone learns your labels, not the visual world: it names what you paid to name and guesses at the rest.
Captions look like a way out, and sometimes they are. Contrastive vision-language pretraining pairs each photo with its caption and learns names well. But captions describe whatever people bothered to mention. They skip depth, exact shape, and where one part ends and the next begins, so caption-trained eyes go soft on dense work like detection and segmentation.
Self-supervised vision features take the other path: train a vision encoder on the pixels alone, with targets the image supplies itself, then reuse that encoder everywhere. A common setup cuts a 224 by 224 photo into 16 by 16 patches, which gives a 14 by 14 grid, 196 patches in all, and hides 147 of them on every step. That is a 75 percent mask ratio, and the encoder that survives it carries shape and texture you never paid to label.
This page is about the features as a product: what trains them, what the numbers mean, and how you reuse them. The pixel-rebuilding mechanics in full detail live in masked autoencoder vision. The general principle behind hiding data from itself lives in self-supervised learning. One idea to bring along: an embedding is a list of numbers placed so closeness means relatedness, and these features are embeddings for image patches and whole images.
Think of It Like This
Revising from notes with most of the page covered
Picture yourself revising for an exam with a sheet of paper covering three quarters of your notes. You read the visible quarter, recite the hidden three quarters from memory, then lift the sheet and mark your errors in red. You repeat that across every page, reshuffling the cover each time. Two things happen: your recitation gets sharper, and you start seeing how ideas connect across pages you never studied directly.
The visible patches are the uncovered lines. The encoder is the understanding you build to recite from so little. The decoder is the recitation itself, and the pixel error is the red ink when you check. When the exam ends, you throw the recitation away and keep the understanding.
Where it stops: your notes stay fixed while you revise, but this training draws a fresh random mask on every pass, so the encoder can't memorize which patches hide. And reciting words rewards a good paraphrase, while rebuilding pixels punishes every wrong shade, which is why these features keep texture that text-trained eyes drop.
How It Actually Works
Here is the full pass in five moves. The diagram above shows it on four patches so every number stays visible; real runs use hundreds.
From photo to patches
Call the image . Chop it into non-overlapping patches, each turned into one token. Here is the patch count: 4 in our small example, 196 in the common 224-pixel setup. Each patch is the content of patch number , summarized as one mean intensity in the example so the arithmetic fits on the page.
Mask, encode, decode
Draw a random mask: a set of patch indices to hide, with the rest left visible. is the hidden set, is the visible set, and the mask ratio is the fraction hidden, 0.75 here. In the example , so patches hide and stays.
Feed only the visible patches into the encoder , a Vision Transformer. Here is the large network whose output you will keep. It returns one feature vector per visible patch plus a summary of the whole image. Then rebuild the full sequence: place a learned mask token, a placeholder vector that says patch missing, into every hidden slot, and run the small decoder over the whole thing. Here is the lightweight network you will throw away. It outputs a prediction , read x-hat sub i, for each hidden patch.
The loss, on masked patches only
Score only the hidden patches. With the true patch content and the prediction:
In plain words: the mean squared error over the hidden set. is the training loss, is the count of hidden patches, and the square punishes a miss of 4 four times harder than a miss of 2. Visible patches pay nothing. That last rule is the whole game: if visible patches counted, the net could coast by copying what it already sees.
Worked example: three predictions, error 8.0
Take four patches with mean intensities . Keep patch 2, the 20, and hide patches 1, 3 and 4, whose true values are . Suppose the decoder predicts .
Errors: , , . Squares: , , , summing to . Divide by :
So this step reports a loss of 8.0, and the gradient pushes all three predictions toward their true values at once. Notice which miss dominates: the 4-point error contributes 16 of the 24, two thirds of the loss. Big misses shout; small ones whisper. The diagram draws these exact four cells and this exact 8.0.
What you keep and how you use it
Training ends, and the decoder goes in the bin. What ships is the encoder : photo in, feature vectors out. You use it two ways. Frozen, you train only a small head on your labels, a linear probe, which tells you what the features already knew. Fine-tuned, you keep training the encoder on your labels at a small step size, which rewires the features for your task and scores higher on dense jobs.
The trade is explicit. You buy spatial detail without labels: features that know edges, texture and part layout because rebuilding demanded them. You pay in naming: with no text seen in training, the encoder can't map a new word to a picture, so open-vocabulary naming wants contrastive vision-language pretraining, and a chatbot that sees wants vision-language models to connect these eyes to words.
The bet underneath is that neighbors predict the missing: natural photos repeat themselves, so a hidden patch resembles its surroundings plus object structure. It fails on noise, on blank walls with nothing to learn, and on objects small enough to hide whole inside the mask. And one warning that saves real projects: these encoders earn their keep after fine-tuning, not as frozen scorers, so judge them by fine-tune numbers.
Code
The snippet below reruns the worked example exactly: three true values, three predictions, one mean.
def masked_mse(true_vals: list[float], pred_vals: list[float]) -> float: """Mean squared error over hidden patches only.""" assert len(true_vals) == len(pred_vals) and len(true_vals) > 0 return sum((t - p) ** 2 for t, p in zip(true_vals, pred_vals)) / len(true_vals)
true_vals = [10.0, 30.0, 40.0] # hidden patches 1, 3, 4pred_vals = [12.0, 28.0, 36.0] # decoder guessesprint(masked_mse(true_vals, pred_vals)) # -> 8.0print([round(p - t, 1) for p, t in zip(pred_vals, true_vals)]) # -> [2.0, -2.0, -4.0]The first line prints the loss from the worked example, 8.0. The second line prints each signed error, matching the hand computation of +2, -2 and -4. Visible patches never enter the function, which is the point: pass them in and you've already broken the rule the loss depends on.
Watch Out For
Scoring visible patches too
Tempting, because more terms feel like more signal. Resist it. Visible patches reconstruct almost for free by copying the input, so including them waters the mean down and the gradient spends itself on the easy terms.
Symptom: the loss reads low from the first epoch while downstream fine-tunes sit flat. Fix: take the mean over only, exactly as the loss above writes it, and structure your code so visible values can never reach the loss function.
Asking a label-free encoder for names
These features never met a word in training, so prompting them with text the way you'd prompt a caption-trained model returns noise, and a frozen probe trails text-grounded encoders on naming tasks. That gap looks like failure and it isn't; it's the wrong test.
Symptom: frozen classification accuracy disappoints while detection fine-tunes shine. Fix: fine-tune the backbone for spatial jobs, and reach for a caption-trained encoder when the job is naming new things with words.
The Quick Version
- Self-supervised vision features are encoder outputs trained on pixels alone, reused frozen behind a small head or fine-tuned for your task.
- Masked pretraining hides about three quarters of each image and rebuilds only the hidden patches, so copying neighbors can't pass.
- The loss is mean squared error over hidden patches: in the worked example, errors of +2, -2 and -4 square to 4, 4 and 16 and average to 8.0.
- The decoder is scaffolding and gets discarded; the encoder is the product you ship.
- These features carry shape and texture for dense work like detection and segmentation, but no word meanings, so open-vocabulary naming needs a caption-trained encoder.
- Judge them by fine-tune scores, never by frozen naming accuracy or by a loss that includes visible patches.