Rotary Position Embeddings
Instead of adding a position vector to every token, rotary embeddings rotate a token's query and key vectors by an angle tied to its position in the sequence.
Why Does This Exist?
Positional-encoding fixes order blindness by adding a position vector to each token embedding before attention runs. That works, yet it bakes in absolute position: token 50 and token 47 each carry their own slot number, and the network must learn to recover the fact that matters, which is that one sits 3 steps behind the other.
The edges hurt most. A model trained on positions up to some length has never shaped its attention patterns around larger angles, so pushing past the trained length degrades quality. Absolute slots are the wrong quantity for the job, and recovering distances from them costs learning effort at every layer.
Rotary embeddings, RoPE, sidestep both issues. Instead of adding a vector, RoPE rotates the query and key vectors themselves by an angle that grows with position. Rotation has a property addition lacks: the angle between two rotated vectors depends only on how far apart their positions are. Relative distance falls out of the geometry for free, exactly the quantity attention scores need. The host mechanism is self-attention, and the decoder companion is causal-masking.
Think of It Like This
Two clock hands holding their gap
Picture two clock hands that both started at 12 and advance at the same steady rate, one starting 6 ticks ahead of the other. Check them at 1 oclock or at 11 oclock: the angle between them is identical both times, because both hands kept ticking forward together and only their gap matters.
RoPE works the same way. Rotate a query by an angle tied to its position and a key by an angle tied to its position, and the angle between the rotated pair depends only on the difference between the positions. Shift both forward together and nothing changes.
Where it stops: clock hands tick at one shared rate while RoPE uses many frequencies at once, fast for some dimension pairs and slow for others. One gap reading also never tells you where on the dial you are, which is why tasks needing absolute position still add it separately.
How It Actually Works
RoPE treats a query or key vector as a run of 2D pairs, dimensions , , and so on, and rotates each pair on its own. For a token at position and a pair with frequency , the rotation applied to that pair is:
Here is the token position, is the fixed frequency for pair , and is the standard 2D rotation by angle . Low index pairs use large and spin quickly with position, high index pairs use small and turn slowly, mirroring the frequency ladder of sinusoidal encodings. The rotation is applied to queries and keys at attention time, right before the dot product.
Why the score depends only on distance
Rotating a query at position by and a key at position by leaves both lengths unchanged and shifts both angles. The dot product of two vectors equals the product of their lengths times the cosine of the angle between them, so it reads only the gap , never or alone. Shift both positions forward by the same amount and the gap, hence the score, stays put. Additive encodings never guaranteed this. They left it for training to approximate.
Only queries and keys rotate
Values carry the content a token contributes once found relevant, so they stay untouched. Rotation shapes only the score between a query and a key. Everything downstream, the scale by and the softmax over the row, proceeds exactly as in ordinary attention. RoPE is a preprocessing step on and per head, not a new attention formula.
Worked example
Fix one frequency and start both vectors at . Rotate the query to position , an angle of radians, and the key to position , an angle of radians. The gap is radians, and the dot product is . Now move both forward: query to position ( radians) and key to position ( radians). The gap is again radians, and the dot product is again . Same distance, same score, regardless of absolute slot. The diagram draws exactly these two cases side by side.
Code
Rotating a toy 2D query and key and confirming the score follows the gap, using only the standard library. The dot of the rotated pair reduces to the cosine of the angle gap, so this checks the gap property directly.
import math
theta = 0.1print(round(math.cos((5 - 2) * theta), 4)) # -> 0.9553print(round(math.cos((15 - 12) * theta), 4)) # -> 0.9553Positions and sit three apart, and so do and . Both prints agree at , which is : the invariance the diagram draws.
Watch Out For
Rotating the values too
The symptom is silently wrong output with perfect shapes. Additive encodings touch the whole embedding, so it feels natural to rotate everything, but RoPE must reach only and . Rotated values blend the wrong content into every output, and nothing in the shapes complains.
Check the implementation at the point queries and keys are built. Values should pass through with no rotation call anywhere near them.
Expecting free generalization past the trained length
The symptom is quality falling off a cliff past the trained context even though distances are relative. Positions beyond the trained range produce angles the network never saw during training, so its learned handling of those angles is guesswork.
Relative structure is exactly why rescaling tricks can stretch the angles after the fact, but that rescale is a deliberate extra step. RoPE alone does not grant it.
The Quick Version
- RoPE rotates query and key vectors by an angle tied to position instead of adding a position vector to the embedding.
- Vectors are handled as 2D pairs, each pair spinning at its own frequency from fast to slow.
- The dot product between rotated pairs reads only the position gap, never the absolute slots, as at gap shows twice.
- Values stay untouched because they carry content, not scores.
- Trained length limits remain. Angles beyond it are unseen, so extension needs an explicit rescale step.