Cross-Modal Search Every Way
Search photos with words, words with photos, clips with sound, and back again. One index serves every direction.
Why Does This Exist?
Single-direction tests hide weak ways: text finds photos while sound finds nothing. Any-to-any eval over four senses exposes gaps. Products need image retrieval plus audio plus clip search from one stack.
This page covers multi-sense search only. Pair training is the alignment page.
The one idea you need first.
Index holds unit vectors from all senses. Query ranks all items by cosine. mAP averages precision over ranks; Recall at counts top- hits. Every ordered pair of senses reports separately.
Think of It Like This
One card catalog for books plus photos plus records
A library files books, photos, and records by subject in one catalog. A tune finds its photo, a photo finds its caption, and a caption finds its film. Missing links show as thin catalog drawers.
Vectors are cards, cosine is subject order, and mAP grades the drawers. Where it stops: cards stay hand-checked, while vectors learn from noisy pairs.
How It Actually Works
Encode each sense once into the joint index. For each ordered pair , rank items per query. Mine hard negatives within scenes. Report mAP plus per direction and the mean. Rerank tops with matchers for fine wins.
In plain words: index everything, query anything, and average all ways.
Worked example
Four senses give directions. Mean percent hides audio-to-image percent. Hard scene pools cut text-to-image from to . The point drop directs the next data buy toward scene-close audio pairs.
Watch Out For
Averaging away the weak direction
Mean scores look fine while one way flops for users. Sound queries fail silently. Report per-direction tables and gate releases on the worst way, not the mean.
The Quick Version
- Cross-modal search spans text, image, video, audio both ways.
- Joint indexes serve every query form.
- mAP plus Recall at K grade each direction.
- Hard scene pools reveal fine gaps.
- Worst-direction scores gate releases.