Foundation Models Overview

Pretrain one large backbone on broad data, then adapt it cheaply to many jobs. Scale plus breadth replace per-task training.

Foundation models pretrain broadly then adapt to many tasks

Why Does This Exist?

Per-task training repeats data cost for every job. Foundation training pays once on broad pairs, then transfer learning adapts with probes or light tuning. This residual page frames the shift; vision specifics live in vision foundation models and use lives in pretrained models.

This idea also grounds scaling talk in scaling laws.

The one idea you need first.

Pretrain learns general features $f$ on broad data $D$ ; adapt maps $f$ to task $t$ with probe $p_t$ or small tune. Emergence means new skills appear past scale without direct teaching.

Think of It Like This

General schooling plus short job training

Students study reading and sums for years, then learn cashier work in a week. Schooling transfers because basics repeat across jobs. Job training stays short and specific.

Broad data is schooling, probes are job weeks, and scale is school years. Where it stops: schools teach intent, while pretraining only predicts data patterns.

How It Actually Works

Train $f$ with contrastive, masked, or caption losses on billions of items. Freeze for probes or tune lightly per task. Evaluate breadth: classification, retrieval, grounding, segmentation from one $f$ . Track data, size, and adapt cost per task.

In plain words: one $f$ , many $p_t$ , and breadth as the report card.

Worked example

Backbone $300$ M trained on $2$ B pairs: linear probe ImageNet $82.1$ percent, retrieval $R@1$ $68$ percent, grounding $60$ percent. Task-tuned detection adds $5$ AP for $4$ percent weights. One pretrain covers three scores, which per-task training would triple in cost.

Watch Out For

Calling every large backbone foundational

Narrow training on one dataset does not transfer broadly despite size. Probes on new tasks sit flat. Check breadth across three task families before the label sticks.

The Quick Version

Foundation models pretrain once on broad data.
Light probes or tuning adapt to each task.
Scale plus breadth drive transfer.
Vision specifics live in the vision child page.
Breadth across tasks proves the claim.