Skip to content
AI360XpertBeta

Vision Transformers

Vision Transformers slice an image into fixed-size patches, treat each one as a token, and run ordinary self-attention over that sequence of image tokens.

A 3 by 3 image becomes nine patch tokens with position tags added, and the whole sequence runs through one ordinary self-attention block with no convolution anywhere
A 3 by 3 image becomes nine patch tokens with position tags added, and the whole sequence runs through one ordinary self-attention block with no convolution anywhere

Why Does This Exist?

For a decade, the answer to how a network reads an image was a convolutional neural network. Convolution earns that spot honestly. A shared kernel scanning an image detects a pattern wherever it appears, and stacked kernels build from edges to textures to whole objects. That is a built-in assumption about images: nearby pixels relate to each other, and a pattern found in one corner means the same thing in another.

The wall appears when you ask whether that assumption is necessary or just convenient. Self-attention already let any element look at any other element, with no locality baked in. It just needed a sequence, and images are not naturally sequences. Vision Transformers, from the original 2020 paper, answer directly: cut the image into patches, treat them as a sequence, and let ordinary self-attention do the rest.

Think of It Like This

Reading a photo as a deck of postcards

Cut a photograph into a 3 by 3 grid of postcard squares, stack them into a deck, and hand it to someone who has only ever read sentences. They never saw a photo laid out flat. They only know how to weigh each word against every other word, near or far.

Tell them where each postcard originally sat, and they apply the same trick to the deck: every postcard weighed against every other postcard, relevance decided by content rather than by grid geometry. That is the whole move. Trade nearby pixels matter more for every patch can matter to every other patch, and the model decides how much.

Where it stops: a reader still reads in order, while self-attention without position tags has no order at all. Shuffle the deck and the reader notices. The model does not, unless you tag each card. That tag is load-bearing, not decoration.

How It Actually Works

From pixels to a sequence: patchify, flatten, project

Split the input into a grid of fixed-size, non-overlapping squares. Sixteen by sixteen pixels is the original paper's default. Flatten each patch's raw pixels into one vector and project it linearly into the model's embedding width. One token per patch. That conversion, a 2D grid of pixels into a 1D sequence of vectors, is exactly the shape self-attention expects, and it is the only new machinery in the whole idea.

Count it on the standard input size. A 224 by 224 image cut into 16 by 16 patches gives (224/16)2=196(224 / 16)^2 = 196 patches on a 14 by 14 grid, each holding 16×16×3=76816 \times 16 \times 3 = 768 numbers before projection. The diagram shows the same arithmetic at 3 by 3 so you can see every patch.

Position tags, because flattening erases layout

Once patches become a flat sequence, nothing records which patch sat where. Self-attention treats its input as a set of vectors, so a learned position tag is added to each patch token before the first attention layer. Skip this step and the model literally cannot distinguish an image from the same patches shuffled. That failure is silent, which is why it gets its own card below.

Ordinary transformer blocks, unchanged

From here everything is standard self-attention: queries, keys and values built per token, every token scored against every other token, weighted sums blended into new representations, stacked across layers. Nothing in the attention computation knows its input started as pixels rather than words. That is the point. A classification head reads out from one designated token, or a pooled summary of all of them, at the end.

What convolution gave for free that attention does not

A convolutional kernel enforces two things structurally, before any training: locality, since a kernel only looks at a small neighbourhood, and translation equivariance, since the same kernel applies at every position. Self-attention imposes neither. Every patch can attend to every other patch from the first layer, and nothing treats patch 3 attending to patch 4 differently from patch 3 attending to patch 40. That flexibility is real power, and it is also why a Vision Transformer needs substantially more training data than a comparably sized CNN to match it. The model must learn from data what convolution assumes for free. Past a large enough data threshold Vision Transformers routinely overtake CNNs, because the assumptions baked into convolution are also a ceiling on what it can represent.

Code

Patchifying a small image into a sequence and confirming the exact token count a real ViT-style split produces.

import numpy as np

def patchify(image: np.ndarray, patch_size: int) -> np.ndarray:    h, w, c = image.shape    grid = h // patch_size    patches = image.reshape(grid, patch_size, grid, patch_size, c)    patches = patches.transpose(0, 2, 1, 3, 4)         # group by (row, col) of patches    return patches.reshape(grid * grid, patch_size * patch_size * c)

rng = np.random.default_rng(0)image = rng.normal(size=(224, 224, 3))                  # a standard ViT input sizepatches = patchify(image, patch_size=16)print(patches.shape)  # -> (196, 768), 196 patches (14x14 grid), each 16*16*3 pixels flattened

196 patches is exactly (224/16)2(224 / 16)^2. Same arithmetic the diagram shows at a smaller scale.

Watch Out For

Training a ViT from scratch on a small dataset and expecting CNN-level accuracy

Self-attention has no built-in locality or translation bias, so a Vision Transformer trained from scratch on a modest dataset routinely loses to a similarly sized CNN. It has not seen enough examples to learn what convolution assumes structurally. That is the direct cost of the flexibility above, not a bug. In practice Vision Transformers are almost always pretrained on large datasets first, or fine-tuned from a checkpoint that already was.

Dropping the position tags and getting shuffle-invariant nonsense

Without position tags the output is invariant to patch order. Feed the same patches scrambled and the model answers identically, which silently destroys the spatial structure the task depends on. The failure shows up as poor accuracy with no obvious cause rather than an error, because the model still runs. If a ViT underperforms for no reason, check that the tags are added before doing anything else.

The Quick Version

  • Vision Transformers cut an image into fixed-size patches, flatten and project each into a token, and run ordinary self-attention over the sequence.
  • A learned position tag restores the spatial layout that flattening into a sequence erases.
  • Self-attention has no built-in locality or translation bias, unlike convolution, so ViTs need more training data to learn what CNNs get structurally for free.
  • Past a large enough data threshold Vision Transformers routinely overtake CNNs, since the assumptions convolution bakes in also cap what it can represent.
  • The only new machinery is the patch-to-token conversion. Everything downstream is a standard transformer.