Image-Text Retrieval Rank

Given a photo find its caption, or given words find the photo, ranked from a large pool. The true mate should sit at rank one.

Why Does This Exist?

Joint spaces are only useful if mates rank first. Retrieval tests both directions on COCO $5000$ and Flickr sets with Recall at $1, 5, 10$ . Dual encoders give fast search; cross-attention reranks the top $128$ . Skill here feeds image retrieval products.

This page covers ranking tests only. Encoder training is the ALIGN and SigLIP pages.

The one idea you need first.

Recall at $K$ is share of queries whose mate lands in top $K$ . Text-to-image and image-to-text run separately. Hard negatives are close non-mates that test fine sense.

Think of It Like This

Finding one library card among thousands fast

A library holds a million cards. A fast index pulls 128 likely cards, then a careful reader orders those by hand. Speed plus care beats either alone.

Dual encoders are the index, cross-attention is the reader, and Recall at $1$ is the grade. Where it stops: library cards stay fixed, while web captions vary in style.

How It Actually Works

Encode all photos $v_i$ and texts $t_j$ once. Score cosine for fast recall. Rerank top candidates with matcher $m(v,t)$ trained on hard negatives. Report $R@1, R@5, R@10$ both ways on COCO and Flickr.

In plain words: index broadly, read narrowly, and count top- $K$ hits.

Worked example

Query photo, $5000$ captions, mate score $0.71$ , best negative $0.69$ . Rank $1$ hit counts for $R@1$ . Over $1000$ queries with $620$ rank-one hits, $R@1 = 62.0$ percent. Reranking lifts $62.0$ to $68.5$ by fixing $65$ near misses, which is the rerank payoff.

Watch Out For

Testing only easy random negatives

Random pools flatter scores near $90$ percent while product queries flop. Hard sets from the same scene cut $15$ points. Mine hard negatives and test scene-close pools before shipping search.

The Quick Version

Retrieval ranks mates both directions over thousands.
Recall at 1, 5, 10 on COCO and Flickr is standard.
Dual encoders index fast; matchers rerank top picks.
Hard negatives reveal true fine sense.
Bidirectional scores both matter.