Fusion Early Late Hybrid

Mix senses at input, at output, or through small bottleneck layers mid-net. Place of mixing sets cost and gain.

Fusion mixes senses early plus late plus in a hybrid middle

Why Does This Exist?

Where senses meet decides both accuracy and bills. Early concat learns rich mixes at high cost; late votes stay cheap but shallow; bottlenecks split the difference. Choice shapes multimodal learning results more than backbone size.

This page compares places only. Alignment training is the alignment page.

The one idea you need first.

Early fuses tokens $[X_{img}, X_{txt}]$ before deep layers. Late fuses scores $y = (y_{img} + y_{txt})/2$ . Hybrid routes through $B$ bottleneck tokens with cross-attention per layer.

Think of It Like This

Cooking together versus plating separately

Early teams chop together from the start and flavors meld, but kitchens crowd. Late teams cook apart and plate together, staying tidy yet shallow. Hybrid stations pass small tasting bowls between teams.

Tokens are ingredients, bottlenecks are tasting bowls, and votes are plates. Where it stops: food mixes physically, while vectors mix by learned weights.

How It Actually Works

Try early concat for top accuracy on short clips, late mean for cheap baselines, and $4$ -token bottlenecks per layer for balanced runs. Dropout whole senses at $0.2$ during training so missing inputs still work. Compare accuracy per FLOP on validation.

In plain words: $B$ sets mixing bandwidth, dropout sets robustness, and FLOP sets the bill.

Worked example

Emotion clips: vision alone $68.2$ percent, audio alone $64.5$ , late mean $71.0$ , early concat $74.3$ , hybrid $74.8$ at $60$ percent of early cost. Hybrid keeps $0.5$ over early while halving FLOPs. The table picks hybrid for deployment.

Watch Out For

Fusion hiding a dead sense

Gains can come from one strong sense while the other idles. Ablations show no drop when weak inputs zero. Always report single-sense plus fused numbers in audio-visual learning tests.

The Quick Version

Early concat mixes tokens for top accuracy at high cost.
Late voting averages scores cheaply but shallowly.
Hybrid bottlenecks balance gain versus FLOPs.
Sense dropout guards missing inputs.
Ablate senses to prove joint use.