RoPE
Rotary Position Embedding is an advanced mathematical technique in transformers that encodes sequence position by physically rotating vectors in a latent space.
Think of It Like This
Like giving two dancers specific instructions on how far to rotate so they can instantly calculate their relative angle to each other at any moment.
Unlike absolute embeddings that just add a static vector, RoPE applies a rotation matrix to the Query and Key vectors based on their position index. The dot product between them then naturally depends only on their relative distance. It provides superior extrapolation to longer sequence lengths compared to older positional methods.