Projector vs. Cross-Attention
A projector translates a picture into extra words pasted at the front of your prompt. Cross-attention instead leaves the picture on the side and lets every layer of the model glance at it.
Why Does This Exist?
Your language model can speak but it cannot see. Your vision encoder can see but it cannot speak. Something has to connect them, and where you put that connection decides what the finished system can handle and what it costs you.
The cheapest fix is a projector: a small network that translates image numbers into word numbers, then pastes them at the front of the prompt. The deeper fix is cross-attention: new layers inside the model that look up image numbers while the text is being written. One leaves the model untouched. The other operates on it.
The scale of the problem is what forces the choice. Every image becomes hundreds of tokens once translated, and those tokens sit in the same limited window as your words. Say your window holds 4,096 tokens and one image tile costs 576 of them. Seven images cost 4,032 tokens and barely fit. The eighth image, at 4,608, overflows. A minute of video would need hundreds of images, so the paste-up-front design runs out of room fast.
This page is about that wiring choice: how each path moves numbers, what it costs, and when each breaks. The full system around the choice lives in vision-language models. The general math of mixing image and text numbers is multimodal fusion. The encoder both paths read from is trained by contrastive vision-language pretraining, and the budget the projector spends is context windows.
The one idea you need first: a token is just a list of numbers, called an embedding, that stands for one chunk of input. Attention is a lookup: a query from your text is compared against keys from some store, the matches become weights that sum to 1, and the output is that weighted mix of the store's values. Both designs below are built from those two moves.
Think of It Like This
The translated note vs the picture on the wall
Suppose you must answer written questions about a photograph. A friend offers you two kinds of help.
With the projector, your friend studies the photo, writes you a detailed note describing it, then takes the photo away. You write your answers using only the note. If the note missed a small street sign in the corner, you will never recover it, and ten photos mean ten notes stacked on your desk before you have written a word.
With cross-attention, your friend pins the photo on the wall beside your desk and leaves it there. You write one sentence, glance up to check a detail, write the next sentence, glance up again. The desk stays clear no matter how many photos are pinned up, but someone had to install the wall, the pins, and your habit of looking up.
Here is where the picture stops. Real models never write English notes to each other. The note is a list of numbers, the glance is a matrix multiply, and neither the note nor the wall is free. The note spends your desk space, which is context. The wall spends construction work, which is surgery on the model plus the training to make the new layers useful.
How It Actually Works
Both paths start in the same place: a vision encoder turns an image into visual embeddings, each a vector of length . Your prompt is text embeddings, each of length . The question is only how those two stacks meet.
The projector path: translate, then prepend
A projector is usually one linear map plus a nonlinearity, sometimes just the linear map. For one visual embedding (a list of numbers), with a weight matrix (a table with rows and columns) and a bias vector (one number per output row), the translated token is:
In plain words: is what the eye saw, and are the learned translator, and is the result rewritten in the language model's own number dialect. Each of the visual embeddings is translated this way, giving tokens of length .
Then you prepend. The model's input becomes one long sequence:
where each is one of your text tokens. The model itself never changes. Its ordinary self-attention reads the whole row, image part and text part together, exactly as if you had typed a very long prompt.
One token through the map, by hand: let and , with holding rows and , bias , and . The first output number is . The second is . So . That is the whole trick, repeated per token. In our running fixture the encoder emits 4 visual tokens and your prompt has 3 text tokens, so the model reads tokens.
The cross-attention path: short sequence, side store, per-layer queries
Cross-attention refuses to prepend. The visual embeddings first pass through a small compressor, often called a resampler, that squeezes any number of them into a fixed count of latents. In our fixture visual tokens become latents, and . Your text stays tokens in the main sequence. The model is then opened up: fresh attention layers are inserted between its existing blocks, and each one queries that side store.
One such layer computes standard scaled dot-product attention. With for queries (built from the current text states), for keys and for values (both built from the visual latents), and for the width of each key:
In plain words: scores every text state against every latent, dividing by keeps those scores from exploding, turns one row of scores into weights that sum to 1, and multiplying by mixes the latents in those proportions.
One query through it, by hand: let , query , keys and . The raw scores are dot products: and . Since , the scaled scores are and . Exponentiate: and . The sum is about , so the weights are and .
Now mix the values and in those proportions: . The query listened mostly to the first latent, a little to the second. Every inserted layer repeats this lookup with its own weights, so the text keeps glancing at the image from the first block to the last.
Worked example: the same 4 tokens take two routes
Keep the fixture fixed and watch it split. Left route: 4 visual tokens of length 3, projector maps each to length 2 (our token becomes ), plus 3 text tokens, so the model consumes a 7-token prompt through untouched layers. Right route: the same 4 tokens compress to 2 latents, the main sequence holds only the 3 text tokens, and each new layer blends the latents at weights into . The diagram draws exactly these numbers, left and right of the divider.
Now the budget that separates them. A common encoder grid slices a 336-pixel tile into patches 14 pixels wide: patches per side, and tokens per tile. Say your window holds 4,096 tokens. Seven tiles cost tokens and barely fit alongside your words. Eight cost and overflow. The projector pays per image out of context. Cross-attention pays per layer out of parameters and training data, and its 2 latents cost the same whether the video is one frame or one hundred.
What each choice costs you
Name the trade plainly. The projector buys the cheapest possible training: the big models stay frozen and you fit a tiny translator on a modest pile of image-text pairs. What it spends instead is context, and context grows with every image you add. Cross-attention buys reach over long media: the sequence stays short because vision lives on the side. What it spends instead is surgery. You add randomly started layers to a trained model and must train them on enough paired data to become good lookups, which is far more data and compute than a projector needs.
So do not reach for one tool always. For a single picture or a handful of them on top of a frozen model, use the projector. It is simple, fast, and honest. For long video, dozens of images, or dense documents where the prompt would overflow, the projector is the wrong tool no matter how you tune it. That is when per-layer lookup earns its cost, or when you must cut tokens per image with tiling or a stronger compressor first.
Each path also rests on an assumption that can fail. The projector assumes a small map can carry visual meaning into word space without loss, and that everything fits in the window. It fails on fine print, tiny objects, and long sequences: detail the translator dropped is gone before the model starts thinking. Cross-attention assumes you can afford the added layers and the data to train them. Starve it and the new layers stay mushy: single-image quality can sit below a cheap projector baseline for a long time, and no prompt trick fixes that because the weakness is in the weights, not the words.
Code
The point of this snippet is the side-by-side check: the projector's translate step and the cross-attention blend, on the exact fixture numbers from above.
import numpy as np
def project(v: np.ndarray) -> np.ndarray: W = np.array([[1.0, 0.0, 1.0], [0.0, 1.0, 0.0]]) # shape (2, 3): dialect map return W @ v # bias is zero here
def attend(q: np.ndarray, K: np.ndarray, V: np.ndarray) -> np.ndarray: scores = (K @ q) / np.sqrt(K.shape[1]) # one score per latent e = np.exp(scores - scores.max()) # subtract max: same weights, no overflow w = e / e.sum() # weights sum to 1 return w @ V
v = np.array([2.0, 1.0, 3.0])print(project(v)) # -> [5. 1.]
q = np.array([1.0, 0.0])K = np.array([[1.0, 0.0], [0.0, 1.0]])V = np.array([[4.0, 0.0], [0.0, 2.0]])print(np.round((np.exp((K @ q) / np.sqrt(2.0)) / np.exp((K @ q) / np.sqrt(2.0)).sum()), 2)) # -> [0.67 0.33]print(np.round(attend(q, K, V), 2)) # -> [2.68 0.66]print(4 + 3) # -> 7The projector prints , the attention weights print , the blend prints , and the prompt length prints . Those are the numbers in the diagram, reproduced by running code rather than retyped.
Watch Out For
Shrinking the image to fit, then wondering why the text vanished
You upload a dense screenshot, the model reads the big headings fine and invents the small print. No error, just confident fiction about numbers it never saw.
The cause sits before either fusion path runs. Most encoders resize every upload to a fixed grid before tokenizing, so fine print is blurred into noise while the token count looks healthy. A projector faithfully translates that blur, and cross-attention faithfully queries it. More prompt budget cannot recover detail the encoder already destroyed.
Check with tiny text first: ask for one small string from the image before trusting any summary. If it fails, tile the image into overlapping crops at the encoder's native size and budget the extra tokens, or raise the per-tile token count with your eyes open about the window math above. Never fix blur by raising temperature or rephrasing.
Adding cross-attention layers with a projector-sized data pile
You insert fresh lookup layers, train on the same small paired set that worked for a projector, and single-image quality drops below the cheap baseline. The loss stalls and the fusion outputs look mushy for epochs.
The cause is the new weights. A projector fits thousands of numbers onto frozen models. Cross-attention layers start uninitialized inside the model and need far more paired examples and steps to become good lookups. Small data trains the translator fine and starves the surgery.
Freeze the backbones, train the fusion layers first on the largest paired data you can gather, and keep the projector baseline at the same data size as your scoreboard. If you cannot grow the data, stay with the projector. New layers without data are a slower model, not a smarter one.
The Quick Version
- A projector translates each visual embedding with and pastes the results at the front of the prompt, leaving the language model itself unchanged.
- Cross-attention keeps the text sequence short and stores vision on the side, and each new layer queries it with .
- In the toy run here, 4 visual tokens plus 3 text tokens make a 7-token prompt on the left, while the right keeps 3 in sequence, compresses 4 tokens into 2 latents, and blends them at weights into .
- Projector training is cheap but every image spends context, so long video overflows the window fast.
- Cross-attention handles long media because the sequence stays short, but you pay with model surgery, data, and compute.
- One picture on a frozen model calls for a projector, while long video or dozens of images call for per-layer lookup or far fewer tokens per image.