RT-DETR Detects in Real Time

RT-DETR picks top encoder spots as queries and fuses scales efficiently, so end-to-end transformers run live without NMS.

Top 300 IoU aware queries plus light fusion pass 50 frames per second

Why Does This Exist?

YOLO runs live but needs NMS tuning. DETR removes NMS but lags on speed and small maps.

RT-DETR bridges them. It follows detr set loss, borrows dino selection, and serves object detection live use.

Think of It Like This

Shortlisting resumes before interviews

You score 10,000 resumes by keyword plus fit, interview the top 300, and run short panels. Full interviews for all would take months.

Encoder spots are resumes, IoU aware scores are fit, and decoder is interviews. Where it stops: bad shortlists cap hires, just as weak selection caps recall.

How It Actually Works

Backbone plus hybrid encoder fuses $P3,P4,P5$ with attention plus RepBlocks. Score each spot by classification times predicted IoU, keep top $K = 300$ . Decode $300$ with deformable attention and set loss. No NMS at export.

Worked example

$80 \times 80 + 40 \times 40 + 20 \times 20 = 6400 + 1600 + 400 = 8400$ spots. Keep $300$ , a $28$ times cut ( $8400/300 = 28$ ). Spot A scores class $0.9 \times$ IoU $0.8 = 0.72$ versus spot B $0.9 \times 0.3 = 0.27$ , so A ranks first despite equal class. Decoder runs $300$ not $8400$ , passing live budgets.

Watch Out For

Selecting by class alone and missing loose boxes

Class-only top-K keeps confident but sloppy boxes. Symptom: high scores with low IoU. Fix: keep IoU aware scoring as in the paper and validate small AP separately.

The Quick Version

RT-DETR keeps NMS free set loss while running live.
Hybrid encoder fuses three scales with light RepBlocks.
Fixture $8400$ spots to $300$ cuts decoder load 28 times.
IoU aware scores rank $0.72$ above $0.27$ despite equal class.
Validate small boxes since selection can starve them.