Multimodal Data Fusion and Alignment

Why Does This Exist?

A product-listing model has photographs, titles, descriptions and a category tree. Every modality obviously helps. The team concatenates image features onto text features, trains, and lands almost exactly where the text-only baseline was.

Two things went wrong, and only one of them is about modelling. The image vector had 2048 dimensions against 300 for text, so the model largely ignored text. And 9 percent of listings had a photo that belonged to a different listing, because an upstream job matched on a key that got reused.

That second problem is the subject. Fusion is the modelling question: where do the modalities meet? Alignment is the data question: are these two things actually about the same thing? Fusion has three or four standard answers you can pick between in an afternoon. Alignment is months of work, and it decides whether any of the answers help.

This page covers the pairing work: the three fusion points, the three alignment problems, and the serving traps. The vectors that meet in the shared space are vector embeddings. The noisy-pair filtering discipline is data quality filtering. The width mismatch that lets one side dominate is a feature scaling problem.

The one idea you need first. Similarity here is a cosine: two unit vectors dotted together, 1.0 for identical directions and lower for everything else. The full definition with the length normalisation is cosine similarity.

Think of It Like This

How It Actually Works

Three places to fuse

Early fusion concatenates the inputs, or lightly processed versions of them, and hands one vector to one model. Cheap to write. Two problems: whichever modality has more dimensions or larger values dominates unless you are careful with scaling, and a missing modality changes the input shape, so there is no graceful degradation.

Late fusion runs one model per modality and combines the predictions, average, weighted vote, or a small model over the outputs. Robust: a missing modality just drops out of the vote. And it cannot learn any interaction, because the modalities never meet before the answer. If the useful signal is the photo shows a red dress and the title says blue, late fusion cannot see it.

Joint or intermediate fusion gives each modality its own encoder, projects both into a shared space, and lets them interact there, concatenation of the projections, or cross-attention where one modality attends over the other. This is what works, and it is what every current multimodal model does.

Alignment is three separate problems

Instance alignment: this image goes with this text. Sounds trivial until you are joining two systems on a key that one of them recycles.

Temporal alignment: this video frame, this audio window and this caption refer to the same instant. Sub-second offsets are invisible to inspection and destructive to training.

Semantic alignment: the two modalities end up in one space where distance means the same thing regardless of which encoder produced the vector. That is what a contrastive objective builds, pull matched pairs together, push mismatched pairs apart, and it is the mechanism behind searching images with a text query.

The data problems that decide the project

Pair quality. Web alt-text is noisy, so a large fraction of image-caption pairs are weakly related or unrelated. Filtering pairs by a similarity threshold from a smaller clean model reliably beats scaling the architecture, for a while. Data quality filtering is the same discipline applied here.

Scale asymmetry. Millions of images and thousands of captions is normal. The rare modality is the bottleneck, and padding it with generated text imports whatever the generator believes.

Missing modalities at serving. This is the normal case, not the exception: a listing with no photo, a call with no transcript yet, a patient record missing the scan. Decide the strategy at training time: modality dropout, randomly drop a modality during training so the model learns to cope, a learned missing-modality token, or a late-fusion fallback path.

One modality dominating. If text alone answers 90 percent of your benchmark, the model will learn to read text and ignore pixels, and your evaluation will call that success. Test with modalities ablated: score text-only, image-only, and both. If both is no better than the best single one, you have not built a multimodal model.

Worked example

Two images and their two captions, each encoder output projected to the same width and normalised to unit length. Image vectors $[1, 0]$ and $[0, 1]$. Caption vectors $[0.8, 0.6]$ and $[0.6, 0.8]$. Each caption vector has length $\sqrt{0.8^2 + 0.6^2} = 1.0$, so a dot product is cosine similarity.

The 2 by 2 similarity matrix comes out as $0.8$ on the diagonal and $0.6$ off it. Concretely, $1 \times 0.8 + 0 \times 0.6 = 0.8$ for the match and $1 \times 0.6 + 0 \times 0.8 = 0.6$ for the mismatch. Matched pairs score higher than mismatched ones by 0.2, and the contrastive objective exists to widen that gap.

Now shift the caption file by one row. Image one is paired with caption two, so the pairs the loss treats as correct are the $0.6$ entries, and it spends the entire training run pushing the $0.8$ entries, the genuinely matched ones, apart. Nothing errors. The loss goes down. You have trained the model to believe the wrong pairing.

Code

1import numpy as np2
3img: np.ndarray = np.array([[1.0, 0.0], [0.0, 1.0]])4txt: np.ndarray = np.array([[0.8, 0.6], [0.6, 0.8]])5
6
7def similarity(a: np.ndarray, b: np.ndarray) -> np.ndarray:8    """Unit vectors, so a dot product is already the cosine."""9    return a @ b.T10
11S: np.ndarray = similarity(img, txt)12print(np.round(S, 2).tolist())  # -> [[0.8, 0.6], [0.6, 0.8]]13print(np.round(np.diag(S), 2).tolist())  # -> [0.8, 0.8]14print(round(float(np.linalg.norm(txt[0])), 4))  # -> 1.015print(round(float(S.diagonal().mean() - 0.6), 2))  # -> 0.2

Swap the diagonal for the off-diagonal and you have the off-by-one bug: same code, same loss curve, opposite lesson.

Watch Out For

The Quick Version

• Fusion is the modelling question, alignment is the data question, and alignment is where the work and the failures are.
• Early fusion concatenates inputs, so the wider modality dominates and a missing one breaks the shape.
• Late fusion combines predictions, survives a missing modality, and can never learn a cross-modal interaction.
• Joint fusion gives each modality an encoder into a shared space where they interact. That is what works.
• Alignment is three problems: instance, temporal, and semantic. A contrastive objective builds the semantic one.
• Filtering noisy pairs beats scaling the model, for a good while.
• Missing modalities at serving are the normal case. Plan for them with modality dropout, an absent token, or a fallback path.
• If one modality answers the benchmark alone, the model will ignore the other and your evaluation will call it success. Score with modalities ablated.
• Matched pairs scored 0.8 and mismatched 0.6. An off-by-one pairing spends training pushing the right pairs apart.

What to Read Next

• Vector Embeddings
• Data Quality Filtering
• Feature Scaling