CvT Mixes Convolutions Into Transformers

CvT uses convolutions to make patch tokens and to project queries and keys, so transformers gain locality without losing global range.

Overlapping conv embedding plus conv projections give locality with global attention

Why Does This Exist?

ViT cuts non-overlapping patches, so edges split and locality must be learned from scratch. That wastes data on small sets.

CvT puts convolutions where they help. It builds on convolutional layers, lifts image classification with fewer photos, and backs object detection pyramids.

Think of It Like This

Tiles laid with overlap plus grout lines

Butt-jointed tiles crack at seams. Laid with overlap and grout, each tile shares edges with neighbors yet the floor still spans the room.

Overlap embedding is the grout, conv projection is shared edges, and attention is the room span. Where it stops: too much grout blurs pattern, just as heavy convs narrow range.

How It Actually Works

Stage 1 embeds with $7 \times 7$ stride $4$ overlapping conv, later stages $3 \times 3$ stride $2$ . Before attention, reshape tokens to 2D and apply depthwise separable $3 \times 3$ conv to queries, keys, values:

Q = DWConv(x)_{s=1}, K = DWConv(x)_{s=2}

$s$ is stride, so keys can subsample. Remove position codes since convs carry geometry. Stack three stages with shrinking maps.

Worked example

$224$ input with $7 \times 7$ stride $4$ gives $(224-7)/4 + 1 = 55.25$ , floored to $56$ with padding, so $56 \times 56 = 3136$ tokens with overlap instead of $14 \times 14$ disjoint. A $3 \times 3$ depthwise on $C = 64$ costs $3 \times 3 \times 64 = 576$ weights versus $64 \times 64 = 4096$ for dense, about $7$ times cheaper while adding locality.

Watch Out For

Keeping position codes that fight conv geometry

Adding learned positions atop conv projections double counts location. Symptom: slower convergence with no gain. Fix: drop position embeddings as the paper does and rely on conv padding for geometry.

The Quick Version

CvT embeds with overlapping convs and projects QKV with depthwise convs.
Overlap keeps edge continuity that disjoint patches split.
Fixture $7 \times 7$ stride $4$ makes 3136 overlapping tokens from 224 input.
Depthwise $3 \times 3$ at width 64 costs 576 weights, far below dense.
Drop position codes since convs already encode geometry.