Vision Foundation Backbones
Three backbone families cover dense maps, name matching, and promptable masks. Pick the family that fits the task.
Why Does This Exist?
One backbone cannot top dense, naming, and masking at once. DINOv2 leads depth and matching, CLIP leads naming and retrieval, SAM leads promptable masks. Choosing by family saves tuning, building on foundation models framing.
This page maps families only. Methods live in DINOv2, zero-shot classification, and promptable pages.
The one idea you need first.
Dense means per-pixel maps; joint means image-text scores; promptable means click or box to mask. Frozen use reads fixed features with small heads.
Think of It Like This
Three lenses for map, name, and trace jobs
A surveyor carries a level for heights, a gazetteer for names, and tracing film for outlines. Each job picks its lens; none covers all three well.
DINOv2 is the level, CLIP the gazetteer, and SAM the film. Where it stops: lenses stay separate glass, while backs can stack in one system.
How It Actually Works
Score candidate backs frozen: DINOv2 patch tokens for depth plus segmentation, CLIP vectors for retrieval plus zero-shot, SAM encoder plus prompts for masks. Pick max per task family. Tune only small heads in pretrained models style.
In plain words: probe three, pick one, and tune lightly.
Worked example
Depth RMSE DINOv2 versus CLIP; retrieval CLIP versus DINOv2; mask IoU SAM versus others. Each family wins its column by plus points, so the table picks backs per job.
Watch Out For
One backbone for every job
Single-pick systems trail specialists points on off-family tasks. Depth from CLIP smears; names from DINOv2 guess. Route per task or stack two backs.
The Quick Version
- Vision backs split into dense, joint, and promptable families.
- DINOv2 leads dense maps and matching.
- CLIP leads naming and retrieval.
- SAM leads promptable masks.
- Probe frozen, then pick per task.