Deformable DETR Samples Sparse Points

Deformable attention samples four points per scale around each query, so multi-scale detection trains in 50 epochs not 500.

Four points per scale replace dense maps so training drops to 50 epochs

Why Does This Exist?

DETR attends $196 \times 3136$ densely, so high-res maps blow up and small objects blur. Training drags to 500 epochs.

Sparse sampling fixes both. It follows detr queries, reuses object detection matching, and feeds later transformer detectors.

Think of It Like This

Inspectors checking four nearby houses

Instead of searching the whole city per case, each inspector checks four houses near the tip address across four district maps. Nearby evidence suffices.

Reference points are addresses, offsets are nearby houses. Where it stops: tips far off mislead locals, just as bad references miss displaced boxes.

How It Actually Works

Each query has reference $p_q$ in $[0,1]^2$ . For $L = 4$ scales and $K = 4$ points, predict offsets $\Delta p$ and weights $A$ summing to 1. Sample $x^l(p_q + \Delta p)$ by bilinear and blend. Cost drops from $O(H^2W^2)$ to $O(NKL)$ .

Worked example

Query at $p = (0.5, 0.5)$ on $28 \times 28$ map. Offsets $(0.02,0.01),(-0.01,0.03),(0.00,-0.02),(0.03,-0.01)$ sample $(0.52,0.51),(0.49,0.53),(0.50,0.48),(0.53,0.49)$ . Weights $[0.4,0.3,0.2,0.1]$ blend values $[0.9,0.7,0.5,0.3]$ to $0.4 \times 0.9 + 0.3 \times 0.7 + 0.2 \times 0.5 + 0.1 \times 0.3 = 0.36 + 0.21 + 0.10 + 0.03 = 0.70$ . Sixteen keys replace thousands.

Watch Out For

Init offsets to zero and stalling refinement

Zero offsets sample one spot repeatedly early. Symptom: slow first 10 epochs. Fix: small random init plus iterative box refinement so references migrate.

The Quick Version

Deformable samples $K = 4$ points per scale, $16$ keys per query.
Reference plus learned offset keeps sampling local and sharp.
Fixture blend $0.36+0.21+0.10+0.03 = 0.70$ from four neighbors.
Training falls from 500 to 50 epochs with better small objects.
Init offsets with spread plus refinement to move references.