Multimodal Fusion
One model that looks at a photo and reads your question at the same time, by turning both into numbers it can mix into a single code.
Why Does This Exist?
You attach a photo and ask “what peak is this?” A text-only model reads the words but never sees the ridgeline. A vision-only classifier might tag “mountain” yet it cannot hold the conversation. You need one decision that uses both at once.
That combining step is multimodal fusion. It turns each sense into numbers, then mixes those numbers into a single code the generator can answer from. Short version: separate eyes and ears, one shared thought.
This page covers that mixing step itself: where to mix, the math that does it, and a worked example you can check. The full systems that ship it are vision-language models. The alignment step that makes image and text numbers comparable is contrastive vision-language pretraining. The choice between pasting visual tokens up front and peeking with attention is projector vs cross-attention.
The one idea you need first is an embedding. An embedding is a short list of numbers that stands in for meaning, so “mountain photo” might become and “what is this?” might become . Fusion starts after you have those two lists. If they live in unrelated spaces, mixing them gives noise, which is why alignment comes first.
Think of It Like This
Tasting soup while smelling it
You lift a spoon of soup. Your tongue reports salt and warmth. Your nose reports thyme and smoke. Neither sense alone is “flavor.” Your brain blends both streams into one impression, then you say “needs salt.”
Map it point for point. Tongue is the text embedding . Nose is the image embedding . Flavor is the fused code . Saying “needs salt” is the generator answering from . Change one stream and the blend shifts: plug your nose and the same soup tastes flat.
Where it stops: your nose and tongue arrived pre-aligned by biology. AI vectors start in different corners of space and must be trained to line up. Fuse before they line up and you mix noise with signal.
How It Actually Works
Call the image embedding in , a list of numbers from the vision encoder. Call the text embedding in , a list of numbers from the text encoder. Here is the vector length, kept at so you can follow it by hand. The fused code is , the single list the generator reads.
Turn each sense into numbers first
An image starts as pixels. A vision encoder maps patches to . Text starts as tokens. A text encoder maps them to . Both encoders must output the same length or you cannot mix them directly. If they do not match, a small learned projection maps one side to the other, which projector vs cross-attention covers in detail.
Three places you can mix
Early fusion concatenates, written , and feeds one long sequence to a joint Transformer. Every text token can attend to every image patch from the first layer. Deepest interaction. Longest sequence, and self-attention cost grows with the square of that length.
Late fusion keeps both towers apart. Each makes its own call, say for “mountain” from vision and from a text-only prior, then a small head combines them, often a plain average . Cheap. Modular. You can swap one tower without retraining the other. But the towers never trade detail while they think, so “the small hut left of the peak” gets lost.
Gated or intermediate fusion splits the difference and is the workhorse. Keep both encoders, then mix their vectors with a learned gate between and :
In plain words: says how much to trust vision on this input. Cross-attention generalizes the same idea: text queries look up image keys and take a weighted sum of image values , so each generated word can peek at a different patch.
Trade-off: depth of interaction against cost and modularity. Early buys nuance and pays in sequence length and joint training. Late buys reuse and pays in missed nuance. Gated buys a per-input balance for a few extra parameters.
Skip fusion when one sense answers alone. If the question is answerable from text, adding pixels adds parameters, data needs, and failure modes for no gain. It assumes and are aligned and comparable in scale. When that fails, one side dominates or the joint space splits by modality, and the generator leans on the language prior while ignoring the photo.
Worked example with a gate of 0.6
Take for a mountain photo and for “what is this?” Let the model set , so it trusts vision a bit more on this question. That is the gate, one number between and .
First coordinate:
Second coordinate:
So . The fused code leans toward but still carries . Compare a plain late average . Close here, yet the average cannot shift weight per question while the gate can. When the next question is text-heavy, the same network can set and lean the other way. That per-input shift is the whole point, and those three numbers match the diagram above and the snippet below.
Code
The point is the check. A gated mix is two lines, and you can see the lean toward vision in the output.
def fuse_gate(v: list[float], t: list[float], g: float) -> list[float]: return [round(g * a + (1.0 - g) * b, 2) for a, b in zip(v, t)]
def late_average(v: list[float], t: list[float]) -> list[float]: return [round((a + b) / 2, 2) for a, b in zip(v, t)]
v = [0.8, 0.2]t = [0.4, 0.6]print(fuse_gate(v, t, 0.6)) # -> [0.64, 0.36]print(late_average(v, t)) # -> [0.6, 0.4]The first line is the worked example: gives . The second line is the late baseline . If you change to , the first print becomes , which is how you confirm the gate actually steers the blend.
Watch Out For
Text steamrolls the image
Your model answers well but barely looks. Swap the photo for a different mountain and the answer stays the same. That is the symptom: per-input image ablations move accuracy by almost nothing, while rephrasing the text moves it a lot.
The cause is prior imbalance. Text gradients arrive early and often, so the gate drifts high for text or the generator learns to answer from alone. Fix it by checking, not hoping. Measure accuracy with blank or swapped images, add modality dropout during training so the model must sometimes answer from vision, and balance batches so image-heavy questions appear. If the gated code barely moves when changes, fusion is decoration.
Averaging scores hides disagreement
Vision says for mountain, text prior says , late average reports . You read that as uncertain. It is not uncertain. It is two confident models disagreeing.
The symptom is a calm on exactly the cases you most need to inspect. The fix: do not average away disagreement. Log and separately, route high-disagreement cases to the joint or gated path, and use averaging only when you have checked that both towers are calibrated on your data. For questions that need fine links like “left of the peak,” late scores cannot express the link at all. Use multimodal RAG or a joint reader when evidence must meet in one context.
The Quick Version
- Multimodal fusion turns an image vector and a text vector into one joint code the generator answers from.
- Early fusion concatenates for deep interaction at quadratic sequence cost. Late fusion averages separate decisions for cheap reuse with missed nuance.
- A gate blends per question as . Here gives from and .
- Fusion assumes aligned, comparable embeddings. Without alignment it mixes noise and the text prior wins.
- Swap the image as a test. If the answer does not move, your model is not looking.