PaliGemma Compact Multitask

Join a small SigLIP encoder to a small Gemma chat model and multitask on captions, QA, and boxes. Three billion weights cover many jobs.

PaliGemma joins a compact SigLIP encoder with a small chat model

Why Does This Exist?

Giant assistants cost too much to fine-tune per task. PaliGemma totals near $3$ B with open weights, so labs adapt it to farms, shelves, and forms. Multitask prefix training keeps caption, QA, and grounding in one checkpoint, feeding visual grounding plus chat.

This page covers the compact mix only. Giant chat tuning is the LLaVA page.

The one idea you need first.

Prefix LM feeds image plus task prefix and predicts the suffix only. Task prefixes like caption or detect switch behavior with no head change. Mix weights set task share.

Think of It Like This

A pocket multitool instead of a toolbox

A 3-kilo toolbox holds every saw, while a pocket tool holds blade, driver, and pliers well enough for daily fixes. You carry it everywhere and swap bits rarely.

Giant models are toolboxes, PaliGemma is the pocket tool, and prefixes are bits. Where it stops: steel bits stay fixed, while prefix vectors train with the model.

How It Actually Works

Encode with SigLIP $400$ M, project to Gemma $2$ B tokens, and train prefix LM on caption plus VQA plus grounding plus OCR batches. Suffix loss only; prefixes condition the job. Fine-tune the mix per deployment with low-rank updates.

In plain words: $3$ B total, four task streams, and prefixes as the switch.

Worked example

Zero-shot caption CIDEr $110$ , VQA accuracy $72$ percent, grounding $58$ percent at $0.5$ IoU. Task-tuned on a shelf set of $5000$ photos, grounding rises to $71$ percent while caption holds $108$ . The $13$ point task gain costs one small fine-tune, which is the compact promise.

Watch Out For

Expecting giant-model fine detail

Small encoders miss tiny text at $224$ pixels and grounding sags on object detection tests. Upscale inputs or tile before blaming the mix.

The Quick Version

PaliGemma totals near 3B with SigLIP plus Gemma.
Prefix LM multitasks on caption, QA, grounding, OCR.
Prefixes switch tasks with no head swap.
Open weights make task tuning cheap.
Resolution limits fine detail, not the mix.