CoAtNet Stacks Conv Then Attention

CoAtNet runs MBConv blocks early for locality then transformer blocks late for range, joining both strengths in one backbone.

Two MBConv stages feed two relative attention stages for local plus global strength

Why Does This Exist?

Pure convs lack range and pure transformers lack bias. Either alone needs more data or depth to match the pair.

CoAtNet orders them. It extends convolutional layers, beats efficientnet efficiency fronts, and informs convnext era hybrids.

Think of It Like This

Masons then surveyors on one crew

Masons lay local bricks fast with tight rules. Surveyors then sight across the whole site for alignment. One crew does both in order.

MBConv blocks are masons, attention blocks are surveyors. Where it stops: surveyors cost more, so you call them only after walls stand.

How It Actually Works

Stages follow C-C-T-T: two MBConv with squeeze-excite, then two transformers with relative position bias $B$ :

y = softmax(QK^T/\sqrt{d} + B)V

$B$ depends on $dx, dy$ offsets, so translation holds. Downsample between stages. Late attention sees $14 \times 14$ maps, not $56 \times 56$ , keeping cost sane.

Worked example

$224$ input gives $112$ , $56$ , $28$ , $14$ maps across four stages. Late attention on $14 \times 14 = 196$ tokens costs $196^2 = 38{,}416$ pairs versus $3136^2 = 9.8$ M early, about $256$ times cheaper. Relative bias adds $2 \times 13 - 1 = 25$ offsets per axis for a $14$ span, tiny tables with large effect.

Watch Out For

Putting attention too early in the stack

Early $56 \times 56$ attention blows memory and learns noisy geometry. Symptom: slow steps with flat accuracy. Fix: keep the C-C-T-T order and downsample before the first transformer.

The Quick Version

CoAtNet stacks two MBConv stages then two relative attention stages.
Relative bias $B$ encodes offsets while keeping translation behavior.
Fixture late $196^2$ pairs cost 256 times less than early $3136^2$ .
Order matters: convs first for bias, attention late for range.
Downsample before attention to hold the budget.