Generalist Vision Agents
Specialists ace one benchmark and fumble the next room. Generalist agents carry skills across tasks and scenes: asked anything, they look, reason, and act with the same eyes.
Why Does This Exist?
Specialists shatter on contact with variety: the detector aces its benchmark and misreads the next warehouse; the QA model answers its dataset and blanks on rephrasings. Deployment is variety — new rooms, new phrasings, new task mixes daily. Retraining specialists per novelty does not scale past a handful of sites.
Generalist agents trade peaks for breadth: instruction-tuned on diverse vision tasks, equipped with tools (detectors, OCR, search, code), acting through the same perception loop whatever the request. The architecture sibling is unified models; the action loop is embodied vision.
The one idea you need first. Instruction tuning trains the agent to map open-ended requests to behaviors across tasks: "count the boxes", "read the serial", "is this safe" — one model, many verbs. Graded on task breadth (worst-case and average over dozens of tasks), not single-task peaks. Tools extend the verbs without retraining: the agent learns when to call the specialist, not how to be one.
Think of It Like This
A locum doctor versus a one-disease specialist
The world's best malaria specialist cures malaria flawlessly and misdiagnoses everything else; posted to a new town, most patients do not have malaria. A good locum handles whatever walks in — less brilliant per disease, dramatically better per town — and refers the hard cases upward with sound judgment.
Generalists are locums: breadth with referral judgment (call the detector tool, escalate to humans). Specialists are the malaria genius: unmatched in-distribution, hazardous out of it. Deployment is always a new town, which decides the hire.
How It Actually Works
Training blends vision-task mixtures (detection, QA, grounding, OCR, episodic memory) formatted as instructions with tool-use trajectories (when to crop, call OCR, run code). The agent perceives, reasons in language or latent steps, acts via tools or embodied controls, and observes results — the ReAct-style loop applied to pixels. Evaluation spans task suites with held-out task types (not just held-out images) to test verb generalization, plus human ratings for open-ended requests where metrics thin out.
Safety layers gate open-ended action: read-only defaults, confirmation for side effects, refusal repertoires for harmful requests.
Worked example
Three deployment tasks; specialist suite versus one generalist (accuracy):
- Task A (specialist's home): specialist , generalist .
- Task B: specialist , generalist .
- Task C: specialist , generalist .
Averages: specialist ; generalist . Minimums: versus . The generalist loses the peak by and wins the average by — and doubles the worst case, which is the number deployment actually meets at a.m.
Code
The point of this snippet is the deployment number: averages deceive, minimums decide.
def deployment_card(scores: list) -> tuple: avg = sum(scores) / len(scores) return round(avg, 3), round(min(scores), 3) # mean and worst case
print(deployment_card([0.95, 0.40, 0.35])) # -> (0.567, 0.35)print(deployment_card([0.80, 0.75, 0.72])) # -> (0.757, 0.72)Watch Out For
Averages that hide the limp
Mean-over-tasks lets a brilliant home task subsidize catastrophic elsewhere: plus two s averages a respectable while the agent fails most rooms. Buyers reading means deploy failures.
Report minimums and per-task tables beside every average; set worst-case floors for deployment. Breadth claims need breadth evidence, not blended evidence.
Open-ended action without guardrails
An agent that can look, click, move, and message will eventually do all four wrongly at once: misread scenes, misdialed tools, real-world side effects. Capability breadth multiplies the harm surface exactly as it multiplies usefulness.
Default to read-only, confirm side effects, sandbox tools, and red-team across task types — not just the demo tasks. The locum needs malpractice insurance before the first shift.
The Quick Version
- Generalists trade single-task peaks for cross-task breadth: instruction tuning plus tools over shared eyes.
- The worked fixture gives generalist average versus , and worst case versus .
- Deployment meets the minimum, so report worst-case floors beside every average.
- Referral judgment (when to call tools or humans) is the core generalist skill.
- Read-only defaults and sandboxed tools gate open-ended action before the first shift.