Skip to content
AI360XpertBeta

Multimodal Fusion

One model that looks at a photo and reads your question at the same time, by turning both into numbers it can mix into a single code.

Two separate embeddings, an image vector and a text vector, fuse into one joint code that the generator answers from.
Two separate embeddings, an image vector and a text vector, fuse into one joint code that the generator answers from.

Why Does This Exist?

You attach a photo and ask “what peak is this?” A text-only model reads the words but never sees the ridgeline. A vision-only classifier might tag “mountain” yet it cannot hold the conversation. You need one decision that uses both at once.

That combining step is multimodal fusion. It turns each sense into numbers, then mixes those numbers into a single code the generator can answer from. Short version: separate eyes and ears, one shared thought.

This page covers that mixing step itself: where to mix, the math that does it, and a worked example you can check. The full systems that ship it are vision-language models. The alignment step that makes image and text numbers comparable is contrastive vision-language pretraining. The choice between pasting visual tokens up front and peeking with attention is projector vs cross-attention.

The one idea you need first is an embedding. An embedding is a short list of numbers that stands in for meaning, so “mountain photo” might become [0.8,0.2][0.8, 0.2] and “what is this?” might become [0.4,0.6][0.4, 0.6]. Fusion starts after you have those two lists. If they live in unrelated spaces, mixing them gives noise, which is why alignment comes first.

Think of It Like This

Tasting soup while smelling it

You lift a spoon of soup. Your tongue reports salt and warmth. Your nose reports thyme and smoke. Neither sense alone is “flavor.” Your brain blends both streams into one impression, then you say “needs salt.”

Map it point for point. Tongue is the text embedding tt. Nose is the image embedding vv. Flavor is the fused code hh. Saying “needs salt” is the generator answering from hh. Change one stream and the blend shifts: plug your nose and the same soup tastes flat.

Where it stops: your nose and tongue arrived pre-aligned by biology. AI vectors start in different corners of space and must be trained to line up. Fuse before they line up and you mix noise with signal.

How It Actually Works

Call the image embedding vv in Rd\mathbb{R}^d, a list of dd numbers from the vision encoder. Call the text embedding tt in Rd\mathbb{R}^d, a list of dd numbers from the text encoder. Here dd is the vector length, kept at d=2d = 2 so you can follow it by hand. The fused code is hh, the single list the generator reads.

Turn each sense into numbers first

An image starts as pixels. A vision encoder maps patches to vv. Text starts as tokens. A text encoder maps them to tt. Both encoders must output the same length dd or you cannot mix them directly. If they do not match, a small learned projection maps one side to the other, which projector vs cross-attention covers in detail.

Three places you can mix

Early fusion concatenates, written [v;t][v; t], and feeds one long sequence to a joint Transformer. Every text token can attend to every image patch from the first layer. Deepest interaction. Longest sequence, and self-attention cost grows with the square of that length.

Late fusion keeps both towers apart. Each makes its own call, say pimage=0.9p_{image} = 0.9 for “mountain” from vision and ptext=0.3p_{text} = 0.3 from a text-only prior, then a small head combines them, often a plain average (0.9+0.3)/2=0.6(0.9 + 0.3) / 2 = 0.6. Cheap. Modular. You can swap one tower without retraining the other. But the towers never trade detail while they think, so “the small hut left of the peak” gets lost.

Gated or intermediate fusion splits the difference and is the workhorse. Keep both encoders, then mix their vectors with a learned gate gg between 00 and 11:

h=g⋅v+(1−g)⋅th = g \cdot v + (1 - g) \cdot t

In plain words: gg says how much to trust vision on this input. Cross-attention generalizes the same idea: text queries QQ look up image keys KK and take a weighted sum of image values VV, so each generated word can peek at a different patch.

Trade-off: depth of interaction against cost and modularity. Early buys nuance and pays in sequence length and joint training. Late buys reuse and pays in missed nuance. Gated buys a per-input balance for a few extra parameters.

Skip fusion when one sense answers alone. If the question is answerable from text, adding pixels adds parameters, data needs, and failure modes for no gain. It assumes vv and tt are aligned and comparable in scale. When that fails, one side dominates or the joint space splits by modality, and the generator leans on the language prior while ignoring the photo.

Worked example with a gate of 0.6

Take v=[0.8,0.2]v = [0.8, 0.2] for a mountain photo and t=[0.4,0.6]t = [0.4, 0.6] for “what is this?” Let the model set g=0.6g = 0.6, so it trusts vision a bit more on this question. That gg is the gate, one number between 00 and 11.

First coordinate:

h1=0.6×0.8+0.4×0.4=0.48+0.16=0.64h_1 = 0.6 \times 0.8 + 0.4 \times 0.4 = 0.48 + 0.16 = 0.64

Second coordinate:

h2=0.6×0.2+0.4×0.6=0.12+0.24=0.36h_2 = 0.6 \times 0.2 + 0.4 \times 0.6 = 0.12 + 0.24 = 0.36

So h=[0.64,0.36]h = [0.64, 0.36]. The fused code leans toward vv but still carries tt. Compare a plain late average (v+t)/2=[0.6,0.4](v + t) / 2 = [0.6, 0.4]. Close here, yet the average cannot shift weight per question while the gate can. When the next question is text-heavy, the same network can set g=0.2g = 0.2 and lean the other way. That per-input shift is the whole point, and those three numbers match the diagram above and the snippet below.

Code

The point is the check. A gated mix is two lines, and you can see the lean toward vision in the output.

def fuse_gate(v: list[float], t: list[float], g: float) -> list[float]:    return [round(g * a + (1.0 - g) * b, 2) for a, b in zip(v, t)]

def late_average(v: list[float], t: list[float]) -> list[float]:    return [round((a + b) / 2, 2) for a, b in zip(v, t)]

v = [0.8, 0.2]t = [0.4, 0.6]print(fuse_gate(v, t, 0.6))  # -> [0.64, 0.36]print(late_average(v, t))    # -> [0.6, 0.4]

The first line is the worked example: 0.6×v+0.4×t0.6 \times v + 0.4 \times t gives [0.64,0.36][0.64, 0.36]. The second line is the late baseline [0.6,0.4][0.6, 0.4]. If you change gg to 0.20.2, the first print becomes [0.48,0.52][0.48, 0.52], which is how you confirm the gate actually steers the blend.

Watch Out For

Text steamrolls the image

Your model answers well but barely looks. Swap the photo for a different mountain and the answer stays the same. That is the symptom: per-input image ablations move accuracy by almost nothing, while rephrasing the text moves it a lot.

The cause is prior imbalance. Text gradients arrive early and often, so the gate drifts high for text or the generator learns to answer from tt alone. Fix it by checking, not hoping. Measure accuracy with blank or swapped images, add modality dropout during training so the model must sometimes answer from vision, and balance batches so image-heavy questions appear. If the gated code hh barely moves when vv changes, fusion is decoration.

Averaging scores hides disagreement

Vision says 0.90.9 for mountain, text prior says 0.10.1, late average reports 0.50.5. You read that as uncertain. It is not uncertain. It is two confident models disagreeing.

The symptom is a calm 0.50.5 on exactly the cases you most need to inspect. The fix: do not average away disagreement. Log pimagep_{image} and ptextp_{text} separately, route high-disagreement cases to the joint or gated path, and use averaging only when you have checked that both towers are calibrated on your data. For questions that need fine links like “left of the peak,” late scores cannot express the link at all. Use multimodal RAG or a joint reader when evidence must meet in one context.

The Quick Version

  • Multimodal fusion turns an image vector vv and a text vector tt into one joint code hh the generator answers from.
  • Early fusion concatenates [v;t][v; t] for deep interaction at quadratic sequence cost. Late fusion averages separate decisions for cheap reuse with missed nuance.
  • A gate gg blends per question as h=g⋅v+(1−g)⋅th = g \cdot v + (1 - g) \cdot t. Here 0.60.6 gives [0.64,0.36][0.64, 0.36] from [0.8,0.2][0.8, 0.2] and [0.4,0.6][0.4, 0.6].
  • Fusion assumes aligned, comparable embeddings. Without alignment it mixes noise and the text prior wins.
  • Swap the image as a test. If the answer does not move, your model is not looking.