Unified Vision Models

Detection, segmentation, captioning, and QA each train separate giants. Unified models do it all with one backbone and task prompts, sharing eyes across every job.

One 300 million parameter backbone replaces five 200 million specialists, sharing eyes at a third of the price

Why Does This Exist?

Five vision tasks mean five $200$ -million-parameter models: a billion parameters relearning edges, textures, and shapes five times, served on five GPUs, updated on five schedules. The waste multiplies with every added task while the shared structure (all five see the same world) goes unexploited. Specialists also fragment knowledge: segmentation's boundaries never help detection's boxes.

Unified models share one backbone across tasks with small task heads or prompts routing the shared eyes per job. Training amortizes, serving consolidates, and related tasks transfer instead of duplicating. The behavior-level sibling is generalist agents; the map is open research.

The one idea you need first. Task prompting tells one model which job to do: a learned token, a text instruction, or a query embedding conditions shared features into task outputs. Detection prompts decode boxes, segmentation prompts decode masks, QA prompts decode answers — same eyes, different questions. New tasks add prompts and heads, never new backbones.

Think of It Like This

A Swiss Army knife versus five toolboxes

Five single-purpose toolboxes hold five near-identical handles with different blades: five weights, five maintenance schedules, five things to lose. A Swiss Army knife shares one handle across every blade — lighter, always pocketed, and each tool benefits from the grip refined by all the others.

Unified backbones are the shared handle: edges and shapes learned once, refined by every task's gradients. Task heads are the blades: small, swappable, cheap. New jobs clip on instead of shipping another toolbox — until the knife grows so thick it opens nothing well, which is the negative-transfer limit below.

How It Actually Works

A shared encoder (transformer over image patches) feeds task decoders selected per input: detection heads predict boxes, mask heads predict segments, text decoders answer and caption. Training samples tasks by ratio (balanced, weighted by difficulty, or curriculum-ordered), and losses scale to comparable magnitudes so no task drowns the rest. Inference routes by prompt: one forward pass per task, or batched multi-task passes sharing the encoder compute.

Routing granularity varies: hard sharing (one encoder, all tasks) is cheapest; mixture-of-experts variants buy task capacity at serving cost.

Worked example

Five tasks, $200$ M-parameter specialists each: $5 \times 200$ M $= 1{,}000$ M $= 1$ B parameters. Unified: $250$ M shared backbone plus $5 \times 10$ M heads $= 250$ M $+ 50$ M $= 300$ M. Ratio: $1{,}000/300 \approx 3.3$ .

A sixth task costs $200$ M from scratch but $10$ M as a new head — $20\times$ cheaper to add. Serving holds one $300$ M model instead of five $200$ M ones: fewer GPUs, one update cadence, shared caches.

Code

The point of this snippet is the amortization: shared trunks plus small heads beat repeated specialists.

1def unified_params(tasks: int, trunk: float = 250.0, head: float = 10.0) -> float:2    return trunk + tasks * head  # millions of parameters3
4
5def specialists(tasks: int, each: float = 200.0) -> float:6    return tasks * each7
8
9print(unified_params(5), specialists(5))  # -> (300.0, 1000.0)10print(round(1000.0 / 300.0, 1))  # -> 3.3

Watch Out For

Tasks that fight in shared weights

Gradients disagree: detection pulls features toward boxes while captioning pulls toward prose, and the shared trunk lands in a compromise serving neither. Negative transfer shows as multi-task scores trailing every specialist — unity at the price of quality.

Balance deliberately: loss scaling, gradient surgery, task-specific adapters that absorb disagreement. If a task trails its specialist badly, fork it a private branch rather than taxing the commons.

Sampling ratios that starve small tasks

Uniform sampling drowns rare tasks; proportional sampling starves them differently (tiny datasets seen once per epoch while giants loop). Either way some head undertrains and the "unified" model ships a weak limb with confident branding.

Tune ratios by validation per task, oversample small-but-important tasks, and report per-task scores always. Averages hide starved limbs; minimums expose them.

The Quick Version

Unified models share one backbone across tasks with prompts and small heads routing per job.
The worked fixture prices $300$ M unified against $1$ B of specialists: $3.3\times$ leaner, $20\times$ cheaper to extend.
Related tasks transfer through shared eyes instead of duplicating edge-and-shape learning.
Disagreeing gradients cause negative transfer; absorb it with adapters or fork private branches.
Tune task sampling per validation scores and report minimums, never just averages.