Core ML

Vision Transformers

Vision Transformers slice an image into fixed-size patches, treat each one as a token, and run ordinary self-attention over that sequence of image tokens.

An image is sliced into a grid of fixed patches, each patch is flattened and projected into a token, and the resulting token sequence runs through an ordinary self-attention block

Why Does This Exist?

For a decade, the answer to "how does a neural network read an image" was, almost without exception, a convolutional neural network. Convolution earns that spot honestly: a shared kernel scanning across an image detects a pattern wherever it appears, and stacking kernels builds up from edges to textures to whole objects. That's not an accident of engineering — it's a built-in assumption about images: that nearby pixels relate to each other, and that a pattern found in one corner means the same thing found in another.

The wall shows up once you ask whether that assumption is actually necessary, or just convenient. Self-attention already had a general-purpose way to let any element look at any other element, with no assumption about spatial locality baked in — it just needed a sequence to operate on, and images aren't naturally a sequence. Vision Transformers, from a 2020 Google paper, answer the question directly: turn an image into a sequence by cutting it into patches, and let ordinary self-attention do the rest, with none of convolution's built-in locality assumption.

Think of It Like This

Reading a photo as a grid of postcards instead of a photo

Imagine cutting a photograph into a 4-by-4 grid of small postcard-sized squares, shuffling them into a deck, and handing the whole deck to someone who has never seen a photograph laid out spatially — they've only ever read sentences, one word after another, and figured out relationships between words no matter how far apart they sit in a page.

Give that reader the deck as a sequence of postcards instead of words, tell them where each postcard originally sat, and they apply the exact same trick: look at every postcard in relation to every other postcard, weighting relevance however the content demands, not however the original grid geometry demands. That's the entire move a Vision Transformer makes — trade "nearby pixels matter more" for "every patch can matter to every other patch, and the model decides how much."

How It Actually Works

From pixels to a sequence: patchify, flatten, project

Split the input image into a grid of fixed-size, non-overlapping squares — 16×16 pixels is the original paper's default. Each patch's raw pixels get flattened into a single vector and linearly projected into the model's embedding dimension, producing one token per patch. Patch embeddings covers this step in full; on this page, the important fact is just that it converts a 2D grid of pixels into a 1D sequence of vectors, exactly the shape self-attention expects.

Position embeddings, because flattening erases layout

Once patches become a flat sequence of tokens, the model has no way to recover which patch sat where in the original grid — self-attention treats a sequence as a set of vectors plus their order, and flattening threw the 2D layout away. A learned position embedding, added to each patch token before the first attention layer, restores that information. Skip this step and a Vision Transformer literally cannot distinguish an image from a version of itself with the patches shuffled.

Ordinary transformer blocks, unchanged

Once the patch sequence has position information, everything downstream is standard self-attention: queries, keys, and values built per token, every token scoring against every other token, weighted sums blended into new token representations, stacked across several layers. Nothing about the attention computation itself knows or cares that its input started life as pixels rather than words — this is the whole point. A classification head reads out from one designated token (or a pooled summary of all of them) at the end.

What convolution gave for free that attention doesn't

A convolutional kernel enforces two things structurally, before any training happens: locality (a kernel only ever looks at a small neighborhood) and translation equivariance (the same kernel, and so the same learned pattern-detector, applies at every position). Self-attention imposes neither. Every patch can attend to every other patch from the very first layer, near or far, and nothing about the mechanism treats "patch 3 attending to patch 4" any differently from "patch 3 attending to patch 40." That flexibility is real power, but it's also the reason Vision Transformers need substantially more training data than a comparably-sized CNN to reach the same accuracy — the model has to learn locality and translation patterns from data that a CNN gets for free from its architecture. Past a large enough training-data threshold, though, Vision Transformers routinely overtake CNNs, because the assumptions convolution bakes in are also a ceiling on what it can represent.

Show Me the Code

Patchifying a small image into a sequence and confirming the exact token count a real ViT-style split would produce.

1import numpy as np2
3
4def patchify(image: np.ndarray, patch_size: int) -> np.ndarray:5    h, w, c = image.shape6    grid = h // patch_size7    patches = image.reshape(grid, patch_size, grid, patch_size, c)8    patches = patches.transpose(0, 2, 1, 3, 4)         # group by (row, col) of patches9    return patches.reshape(grid * grid, patch_size * patch_size * c)10
11
12rng = np.random.default_rng(0)13image = rng.normal(size=(224, 224, 3))                  # a standard ViT input size14patches = patchify(image, patch_size=16)15print(patches.shape)  # -> (196, 768) -- 196 patches (14x14 grid), each 16*16*3 pixels flattened

196 patches is exactly (224 / 16) ** 2 — the same arithmetic the diagram's grid represents at a smaller scale.

Watch Out For

Training a ViT from scratch on a small dataset and expecting CNN-level accuracy

Because self-attention has no built-in locality or translation-equivariance bias, a Vision Transformer trained from scratch on a modestly sized dataset routinely underperforms a similarly sized CNN — it hasn't seen enough examples to learn what convolution assumes structurally. This isn't a bug in the model; it's the direct cost of the flexibility described above. In practice, Vision Transformers are almost always pretrained on large datasets first, or fine-tuned from a checkpoint that already was.

Forgetting the position embedding and getting shuffle-invariant nonsense

Drop the position embedding, and the model's output becomes invariant to patch order — feed it the same patches in a scrambled sequence and, absent position information, it produces the same output. That silently destroys spatial structure the task almost certainly depends on, and the failure often shows up as poor accuracy with no obvious cause, rather than an error, because the model still runs.

The Quick Version

Vision Transformers cut an image into fixed-size patches, flatten and project each into a token, and run ordinary self-attention over the resulting sequence.
A learned position embedding restores the spatial layout that flattening into a sequence erases.
Self-attention has no built-in locality or translation-equivariance bias, unlike convolution — so ViTs need more training data to learn what CNNs get structurally for free.
Past a large enough data threshold, Vision Transformers routinely overtake CNNs, since the assumptions convolution bakes in also cap what it can represent.
The patch-to-token step is a separate, reusable idea, covered on its own page as patch embeddings.