Hybrids Join CNNs With Transformers

Hybrids run convolutions early for bias and transformers late for range, beating either alone on data and compute fronts.

Convs on 56 maps then attention on 14 maps cuts cost 16 times

Why Does This Exist?

Pure CNNs miss far context and pure transformers need huge data. Hybrids cover both with staging.

This page links coatnet, cvt, and convolutional layers as components, not reteaching them.

Think of It Like This

Local guides plus satellite view

Guides walk streets for detail while satellites map districts. One trip uses both in order.

Convs are guides, transformers are satellites. Where it stops: swapping order wastes money, just as early attention wastes compute.

How It Actually Works

Run MBConv or conv embeddings on $56$ and $28$ maps. Switch to attention on $14$ and $7$ maps with relative bias or windows. Fuse or pyramid for heads. Residual umbrella only; details live in component pages.

Worked example

$56 \times 56 = 3136$ conv tokens cost linear $3136 \times 3 \times 3 \times 64 \approx 1.8$ M mults. Late $14 \times 14 = 196$ attention costs $196^2 = 38$ K pairs, $16$ times less than $56$ attention $9.8$ M. Pair keeps bias early and range late.

Watch Out For

Re-teaching components instead of linking them

Copying CoAtNet plus CvT details here triples maintenance. Symptom: drift when papers update. Fix: keep this residual and link components for mechanisms.

The Quick Version

Hybrids stage convs early and attention late for bias plus range.
Early $3136$ convs stay linear while late $196$ attention stays cheap.
Fixture late pairs cost 16 times less than early full.
This is a residual umbrella; mechanisms live in components.
Keep order convs first, attention after shrinking.