Positional Codes Keep Patches Placed

Positional codes add where to what, so permutation-blind attention still knows top-left from bottom-right.

Sine and cosine of rows and columns tag each patch so order matters

Why Does This Exist?

Attention treats tokens as a bag. Shuffle $196$ patches and scores stay identical, but a cat scrambled is not a cat.

Codes restore order. They ground attention mechanisms, replace convolution geometry in convolutional layers stacks, and shape image classification maps.

Think of It Like This

Seat numbers on exam papers

Identical answer sheets need seat numbers to map to students. Content matches but seats decide identity.

Tokens are sheets, codes are seat numbers. Where it stops: wrong seat numbers misassign good answers, just as wrong codes misplace patches.

How It Actually Works

For patch $(r,c)$ and dim $2k$ , 2D sine codes use $PE(r,c,2k) = \sin(r/10000^{2k/d}) + \sin(c/10000^{2k/d})$ with cosine on odd dims, where $d$ is width. Add to token $t + PE$ . Learned codes train a table. Relative bias adds $B(dx,dy)$ inside attention instead.

Worked example

$d = 4$ , row $r = 1$ , col $c = 2$ . Dim 0 uses $1/10000^{0} = 1$ , so $\sin(1) = 0.841$ plus $\sin(2) = 0.909$ for $1.750$ . Dim 1 cosine gives $\cos(1) = 0.540$ plus $\cos(2) = -0.416$ for $0.124$ . Token $[0.5, 0.5]$ becomes $[2.25, 0.62]$ . Shuffle rows and the sums change, so attention follows place.

Watch Out For

Interpolating codes badly on larger tests

Train at $224$ then test at $384$ without resizing codes and maps smear. Symptom: accuracy drops on larger inputs only. Fix: bicubic-interpolate position tables to test grids before inference.

The Quick Version

Attention is order blind, so codes add place to content.
Sine plus cosine of row and column tag each patch deterministically.
Fixture token shifts from 0.5 to 2.25 after adding codes.
Learned tables work, relative bias generalizes better.
Resize codes when testing larger than training.