Skip to content
AI360XpertBeta

Rotary Position Embeddings

Instead of adding a position vector to every token, rotary embeddings rotate a token's query and key vectors by an angle tied to its position in the sequence.

Two vectors rotated by 0.2 and 0.5 radians sit 0.3 apart with dot 0.9553, and rotating by 1.2 and 1.5 keeps the same 0.3 gap and the same 0.9553 score
Two vectors rotated by 0.2 and 0.5 radians sit 0.3 apart with dot 0.9553, and rotating by 1.2 and 1.5 keeps the same 0.3 gap and the same 0.9553 score

Why Does This Exist?

Positional-encoding fixes order blindness by adding a position vector to each token embedding before attention runs. That works, yet it bakes in absolute position: token 50 and token 47 each carry their own slot number, and the network must learn to recover the fact that matters, which is that one sits 3 steps behind the other.

The edges hurt most. A model trained on positions up to some length has never shaped its attention patterns around larger angles, so pushing past the trained length degrades quality. Absolute slots are the wrong quantity for the job, and recovering distances from them costs learning effort at every layer.

Rotary embeddings, RoPE, sidestep both issues. Instead of adding a vector, RoPE rotates the query and key vectors themselves by an angle that grows with position. Rotation has a property addition lacks: the angle between two rotated vectors depends only on how far apart their positions are. Relative distance falls out of the geometry for free, exactly the quantity attention scores need. The host mechanism is self-attention, and the decoder companion is causal-masking.

Think of It Like This

Two clock hands holding their gap

Picture two clock hands that both started at 12 and advance at the same steady rate, one starting 6 ticks ahead of the other. Check them at 1 oclock or at 11 oclock: the angle between them is identical both times, because both hands kept ticking forward together and only their gap matters.

RoPE works the same way. Rotate a query by an angle tied to its position and a key by an angle tied to its position, and the angle between the rotated pair depends only on the difference between the positions. Shift both forward together and nothing changes.

Where it stops: clock hands tick at one shared rate while RoPE uses many frequencies at once, fast for some dimension pairs and slow for others. One gap reading also never tells you where on the dial you are, which is why tasks needing absolute position still add it separately.

How It Actually Works

RoPE treats a query or key vector as a run of 2D pairs, dimensions (0,1)(0, 1), (2,3)(2, 3), and so on, and rotates each pair on its own. For a token at position mm and a pair with frequency θi\theta_i, the rotation applied to that pair is:

R(m,i)=(cos⁡(mθi)−sin⁡(mθi)sin⁡(mθi)cos⁡(mθi))R(m, i) = \begin{pmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \\ \sin(m\theta_i) & \cos(m\theta_i) \end{pmatrix}

Here mm is the token position, θi\theta_i is the fixed frequency for pair ii, and R(m,i)R(m, i) is the standard 2D rotation by angle mθim\theta_i. Low index pairs use large θi\theta_i and spin quickly with position, high index pairs use small θi\theta_i and turn slowly, mirroring the frequency ladder of sinusoidal encodings. The rotation is applied to queries and keys at attention time, right before the dot product.

Why the score depends only on distance

Rotating a query qq at position mm by mθm\theta and a key kk at position nn by nθn\theta leaves both lengths unchanged and shifts both angles. The dot product of two vectors equals the product of their lengths times the cosine of the angle between them, so it reads only the gap (m−n)θ(m - n)\theta, never mm or nn alone. Shift both positions forward by the same amount and the gap, hence the score, stays put. Additive encodings never guaranteed this. They left it for training to approximate.

Only queries and keys rotate

Values carry the content a token contributes once found relevant, so they stay untouched. Rotation shapes only the score between a query and a key. Everything downstream, the scale by dk\sqrt{d_k} and the softmax over the row, proceeds exactly as in ordinary attention. RoPE is a preprocessing step on QQ and KK per head, not a new attention formula.

Worked example

Fix one frequency θ=0.1\theta = 0.1 and start both vectors at [1.0,0.0][1.0, 0.0]. Rotate the query to position 22, an angle of 0.20.2 radians, and the key to position 55, an angle of 0.50.5 radians. The gap is 0.5−0.2=0.30.5 - 0.2 = 0.3 radians, and the dot product is cos⁡(0.3)=0.9553\cos(0.3) = 0.9553. Now move both forward: query to position 1212 (1.21.2 radians) and key to position 1515 (1.51.5 radians). The gap is again 1.5−1.2=0.31.5 - 1.2 = 0.3 radians, and the dot product is again 0.95530.9553. Same distance, same score, regardless of absolute slot. The diagram draws exactly these two cases side by side.

Code

Rotating a toy 2D query and key and confirming the score follows the gap, using only the standard library. The dot of the rotated pair reduces to the cosine of the angle gap, so this checks the gap property directly.

import math
theta = 0.1print(round(math.cos((5 - 2) * theta), 4))  # -> 0.9553print(round(math.cos((15 - 12) * theta), 4))  # -> 0.9553

Positions 22 and 55 sit three apart, and so do 1212 and 1515. Both prints agree at 0.95530.9553, which is cos⁡(0.3)\cos(0.3): the invariance the diagram draws.

Watch Out For

Rotating the values too

The symptom is silently wrong output with perfect shapes. Additive encodings touch the whole embedding, so it feels natural to rotate everything, but RoPE must reach only QQ and KK. Rotated values blend the wrong content into every output, and nothing in the shapes complains.

Check the implementation at the point queries and keys are built. Values should pass through with no rotation call anywhere near them.

Expecting free generalization past the trained length

The symptom is quality falling off a cliff past the trained context even though distances are relative. Positions beyond the trained range produce angles the network never saw during training, so its learned handling of those angles is guesswork.

Relative structure is exactly why rescaling tricks can stretch the angles after the fact, but that rescale is a deliberate extra step. RoPE alone does not grant it.

The Quick Version

  • RoPE rotates query and key vectors by an angle tied to position instead of adding a position vector to the embedding.
  • Vectors are handled as 2D pairs, each pair spinning at its own frequency from fast to slow.
  • The dot product between rotated pairs reads only the position gap, never the absolute slots, as 0.95530.9553 at gap 33 shows twice.
  • Values stay untouched because they carry content, not scores.
  • Trained length limits remain. Angles beyond it are unseen, so extension needs an explicit rescale step.