Gen AI

Projector vs. Cross-Attention

There are two ways to give an LLM vision: either translate the image into 'words' and paste them at the front of the prompt (Projector), or surgically rewire the LLM's brain so it can look at the image during every step of its thought process (Cross-Attention).

Projectors append visual tokens to the text input prompt, while Cross-Attention injects visual context into every single layer of the Language Model.

Why Does This Exist?

When building a Vision-Language Model (VLM), everyone agrees that you need a Vision Encoder to "look" at the image and a Language Model (LLM) to "speak." But the biggest architectural debate in Multimodal AI is exactly how you connect them. Where does the fusion happen?

There are two primary schools of thought:

The Projector Approach (Early Fusion): Used by LLaVA and most open-source models. It is incredibly cheap, fast to train, and treats the image just like text.
The Cross-Attention Approach (Intermediate Fusion): Used by DeepMind's Flamingo and many closed-source frontier models. It requires surgically altering the LLM, making it much harder to train, but allows the model to process complex videos and massive numbers of high-resolution images.

Think of It Like This

The Open-Book Test

Imagine an LLM taking a test where it has to analyze a complex diagram.

The Projector Approach: Before the test starts, the LLM reads a highly detailed, 500-word text description of the diagram. The diagram is then taken away, and the LLM must answer the questions based purely on the text description it just read.
The Cross-Attention Approach: The LLM takes the test with the diagram sitting right next to it on the desk. Every time it writes a sentence, it can glance back at the diagram (Cross-Attention) to double-check specific details.

1. The Projector Approach (LLaVA)

This is the simplest and most popular method today. We take the image and pass it through CLIP, getting 256 "visual embeddings." We pass these through a simple Linear Projector to translate them into the LLM's language space. We then literally just append these 256 visual tokens to the front of the user's text prompt.

If the user typed: "What color is the car?" The actual prompt the LLM sees is: [Visual Token 1] [Visual Token 2] ... [Visual Token 256] "What color is the car?"

Pros:

The LLM's architecture is completely untouched. You don't change a single line of the LLM's code.
Training is incredibly cheap. You only train the tiny Projector network.

Cons:

Context Window Bloat: If you want to analyze a 1-minute video at 10 frames per second, that is 600 images. $600 \times 256 \text{ tokens} = 153,600 \text{ tokens}$ . Your LLM will run out of memory instantly just trying to hold the visual tokens.

2. The Cross-Attention Approach (Flamingo)

This approach is much more complex. We do not append the visual tokens to the text prompt. Instead, we pass the visual embeddings into a Perceiver Resampler. The Perceiver is a special network that forcibly squashes any number of visual tokens (whether it's 256 tokens from one image, or 100,000 tokens from a long video) down into a fixed size, say, 64 tokens.

Then, we surgically open up the LLM. Between every standard Self-Attention layer in the LLM, we insert a brand-new Cross-Attention layer. When the LLM is generating the text response, it uses these Cross-Attention layers to query the 64 squashed visual tokens.

Pros:

Infinite Media Length: Because the visual tokens are injected via Cross-Attention rather than filling up the text context window, these models can analyze massively long videos or thousands of images easily.

Cons:

Training Cost: Because you are injecting brand-new neural network layers into the heart of the LLM, you have to train those new layers. Training an LLM is astronomically expensive compared to just training a tiny Projector on the outside.

Show Me the Code

This pseudocode highlights the structural difference in how the LLM handles the visual data.

1import torch2
3def projector_forward(projector, llm, vision_tokens, text_tokens):4    """5    Projector Approach: Append visual tokens to the text sequence.6    """7    # Translate the vision tokens8    translated_vision = projector(vision_tokens)9    10    # Concatenate sequence: [Vision, Vision, Vision, Text, Text]11    combined_sequence = torch.cat([translated_vision, text_tokens], dim=1)12    13    # The LLM processes everything in a single, standard pass14    output = llm(combined_sequence)15    return output16
17def cross_attention_forward(perceiver, llm, vision_tokens, text_tokens):18    """19    Cross-Attention Approach: Visual tokens sit on the side and are queried.20    """21    # Squash the vision tokens down to a manageable, fixed size22    squashed_vision_context = perceiver(vision_tokens)23    24    # The LLM ONLY processes the text tokens in its main sequence.25    # Inside the LLM, the Cross-Attention layers will query the squashed_vision_context.26    output = llm(text_tokens, cross_attention_context=squashed_vision_context)27    return output

Watch Out For

The High-Res Problem

Both architectures struggle with high-resolution images. If you upload a 4K image of a dense spreadsheet, CLIP will aggressively downsize it to $336 \times 336$ pixels before it even generates the visual tokens. All the text on the spreadsheet becomes blurry unreadable noise. To fix this, modern Projector models (like LLaVA-1.5) chop the 4K image into dozens of smaller tiles, process each tile through CLIP separately, and feed thousands of visual tokens into the LLM, which massively spikes memory usage.

The Quick Version

The Projector Approach (LLaVA) translates the image into tokens and appends them directly to the user's text prompt. It is cheap and doesn't require altering the LLM, but it quickly overwhelms the LLM's context window when analyzing videos or multiple images.
The Cross-Attention Approach (Flamingo) squashes the visual tokens down and holds them outside the main text sequence. The LLM is modified to "peek" at these visual tokens during every layer of its computation.
Cross-Attention is much harder and more expensive to train, but it is necessary for building models that can natively understand long-form video or massive collections of documents.

Related concepts

Whisper Architecture