LLaVA Visual Chat Tuning

Plug CLIP patches into a chat model through one matrix and tune on machine-made instructions. The bot then chats about photos.

LLaVA links CLIP patches to a chat model with one projection

Why Does This Exist?

Chat models read text but not pixels. LLaVA adds vision with the smallest possible joint: a frozen CLIP encoder, one projection matrix $W$ , and a chat LLM. Stage one aligns $W$ on captions; stage two tunes on $158000$ instruction pairs. The result chats about photos, grounding answers in image captioning skill.

This page covers projection plus instruction tuning only. Resampler bridges are the Flamingo page.

The one idea you need first.

Projection $h = W v$ maps each CLIP patch $v$ into LLM word space. Stage one trains only $W$ ; stage two trains $W$ plus the LLM on instruction replies. Instructions are triples of photo, order, and answer.

Think of It Like This

A phrasebook plus role-play lessons

A traveler gets a phrasebook mapping foreign picture cards to home words, then takes 158000 role-play lessons ordering food and asking directions. The book is thin, but practice makes chat smooth.

$W$ is the phrasebook, CLIP cards are pictures, and instruction pairs are lessons. Where it stops: phrasebooks stay fixed in print, while $W$ keeps learning in stage two.

How It Actually Works

Encode photo to patches $V$ with frozen CLIP. Project to $H = W V$ and prepend to instruction tokens. Train next-token loss on answers only. Stage one uses $595000$ captions for $W$ ; stage two uses $158000$ instructions for $W$ plus LLM.

In plain words: captions teach the mapping, instructions teach the manners, and answers alone carry loss.

Worked example

One patch $v = [0.6, 0.8]$ , $W = [[1, 0.5],[0.5, 1]]$ . Then $h = [0.6+0.4, 0.3+0.8] = [1.0, 1.1]$ . Fed with order tokens, the LLM gives answer loss $2.4$ nats early falling to $1.1$ after tuning. The drop measures chat fit, since $W$ shapes every visual token.

Watch Out For

Trusting machine-made instructions blindly

The $158000$ pairs come from a text model that never saw the photos, so some answers invent objects. Bots then hallucinate with confidence. Filter by matching scores and probe on image retrieval before release.

The Quick Version

LLaVA maps CLIP patches into a chat model with one matrix.
Stage one aligns the matrix on captions.
Stage two tunes on 158000 instruction pairs.
Answers alone carry the loss.
Machine-made data needs filtering or hallucinations grow.