Vision-Language Models (VLMs)
A vision-language model gives a chat model sight by translating photo patches into word-like tokens, so one model reads your picture and your question together.
Why Does This Exist?
A chat model can argue about anything you type, yet a photo of a sick plant leaf leaves it with nothing to read. A vision encoder has the mirror problem: it sees the leaf in fine detail and cannot answer a single question. Real questions mix both senses, like "what is wrong with this plant, and how do I treat it," so you need one system that sees and talks at once.
Training that combined system from scratch would mean teaching sight and language together, which costs far more paired data and compute than most teams can spend. The cheaper move is reuse: take a vision encoder that already learned to see, take a chat model that already learned to talk, and train only the small bridge between them.
This page covers that bridge: the three parts, the one equation that translates sight into words, and a worked example you can check by hand. The training that taught the encoder to see lives in contrastive vision-language pretraining. The choice between bridge designs lives in projector vs cross-attention. The question-answering tasks that test the finished system live in visual question answering.
One idea first, because everything below stands on it. An embedding is a list of numbers standing for one chunk of input, placed so related chunks sit near each other. A token is one such chunk. This page turns picture chunks into word chunks, then feeds them to the model as a single sequence.
Think of It Like This
The radio host and the friend at the market
You host a live radio cooking show. You are a sharp talker, but you are blind, and the market is across town. Your friend stands at the stall: she sees every tomato, but she speaks only Spanish, which you do not follow. So you hire a bilingual note-taker. She listens to your friend, writes short English notes, and hands them to you. You read the notes on air and answer the callers.
The friend is the vision encoder, the note-taker is the projector, and you are the language model. The show works because nobody had to learn a new job: seeing, translating, and talking each stay with whoever already does them well.
Here is where the studio breaks down. Real notes are not English sentences but lists of numbers, and anything the note-taker leaves out is gone for good. You cannot ask your friend to look closer. If the note says peppers when the stall holds chilies, you will announce peppers with total confidence, and no caller can tell from your voice that you never saw the stall.
How It Actually Works
A vision-language model is three pieces with one learned joint. A vision encoder turns pixels into vectors. A projector translates those vectors into the chat model's dialect. The chat model reads the translation beside your words and predicts the answer one token at a time.
The three parts
Start with the eyes. A vision encoder slices the image into patches and turns each patch into a visual embedding: a vector of length , where is just how many numbers each patch gets. With patches, where is the patch count, the photo becomes vectors. In our running fixture and . Most builds keep this encoder frozen, its weights untouched, because it already learned to see.
Those vectors mean nothing to the chat model, whose embeddings live in a different space of width , where is how many numbers each word token gets. The projector is the translator: usually one weight matrix with rows and columns, plus one bias vector with a number per row. For a single visual vector , where is what the eye saw for one patch, the translated token is
In plain words: and are the learned translator, and is the same content rewritten in word numbers. Every patch crosses this map on its own, so visual vectors become word-like tokens.
Then comes the assembly. Your prompt is text embeddings through , where is simply how many text tokens you typed and each is one of them. The model input is the concatenation
where is one long sequence the model reads left to right. The chat model itself usually stays frozen too, and it answers with ordinary next-token prediction: predict one token, append it, repeat. Nothing inside the model knows which tokens came from a camera.
Worked example: one patch vector becomes one token
Keep and . Let the translator be rows and , with bias , and let one patch vector be . Multiply row by column and add the bias:
First output number: .
Second output number: .
So . One patch in, one word-like token out, and the diagram draws exactly this crossing. Run all four patch vectors through the same map and you hold 4 translated tokens. Add a 3-token question and the model reads tokens, which is the seven-chip sequence in the picture and in the code below.
How the bridge learns
Only the projector studies. Training shows the frozen pair an image plus a question, the model predicts the answer tokens, and the usual next-token loss sends gradients back to and alone. Because the giants never move, a modest pile of paired examples can be enough, and that is the whole economic point.
The assumption is blunt: the pairs must be right, and the small map must be wide enough to carry what matters. Sloppy pairs teach wrong translations, and no prompt trick later fixes a translator that merged two things it should have kept apart.
What it costs and when to skip it
Name the trade. The projector buys the cheapest possible training and pays in context: every image appends its tokens to the same window your words share. A few photos fit. Dozens of images, long video, or dense pages do not, and shrinking the image to fit destroys the very detail you asked about.
So do not use a bare projector for dense scanned pages. Tiling the page at readable resolution plus a document pipeline, as in document understanding, keeps small print alive. For long media, a different wiring that queries vision from the side earns its keep, which is the comparison in projector vs cross-attention. Watch the shared budget in context windows before you add a second image.
Code
The snippet below runs the worked example: one vector through the projector, then the assembly of the full 7-token input.
import numpy as np
def project(v: np.ndarray) -> np.ndarray: W = np.array([[2.0, 0.0, 1.0], [1.0, 1.0, 0.0]]) # shape (2, 3) b = np.zeros(2) return W @ v + b
v = np.array([1.0, 1.0, 2.0])h = project(v)print(h) # -> [4. 2.]
visual = np.tile(h, (4, 1)) # 4 translated patch tokens, shape (4, 2)text = np.zeros((3, 2)) # 3 text tokens stand in for the questionX = np.concatenate([visual, text], axis=0)print(X.shape) # -> (7, 2)The projector prints , exactly the hand computation, and the assembled input has shape : 7 tokens of width 2, matching the sequence in the diagram.
Watch Out For
Small print turns into confident fiction
You upload a receipt and ask for the total. The model names the shop correctly and invents a total with total confidence. Big shapes survived; the digits did not.
The cause sits before the projector runs. Most encoders resize every upload to a fixed grid, so fine text blurs into noise while the token count looks healthy. The projector faithfully translates that blur, and the chat model fills the gap with a plausible number.
Ask for one tiny string from the image before you trust any summary of it. If that fails, tile the image into overlapping crops at the encoder's native size and budget the extra tokens, or hand dense pages to the pipeline in document understanding. Never raise the temperature or rephrase your way out of blur.
The chat model's habits outvote the picture
Show the model something its training rarely paired, like a blue strawberry, and it can still answer red. The visual evidence says blue, but the language prior, built over countless sentences about red strawberries, shouts louder.
The symptom is specific: answers stay fluent and plausible yet contradict details anyone looking would catch, worst on unusual images. It looks like a vision failure, but the weakness is the frozen talker ignoring good evidence.
Test with at least one image whose answer fights the stereotype before you ship. Prompt for evidence first ("list what you see, then answer"), and keep the grounding habits in hallucination mitigation. When the picture must win, verify on strange inputs, not friendly ones.
The Quick Version
- A vision-language model joins a frozen vision encoder to a frozen chat model with a small trained projector between them.
- The projector translates each patch vector with , so the chat model reads pictures as word-like tokens.
- In the worked example, crosses a 2-by-3 map to become , and 4 visual plus 3 text tokens make one 7-token input.
- Only the bridge trains, using the ordinary next-token loss on image-plus-question pairs, which keeps the bill small.
- Every image spends context, so a bare projector fits a few photos but not long video or dense pages.
- Blurred small print and language habits overruling odd images are the two failures to test before shipping.