Positional Encoding

Self-attention blends values with no sense of order, so each token gets a sine and cosine mark that says where it sits.

Position 1 with width 4 reads 0.84, 0.54, 0.01 and 1.00 from two sine and cosine pairs, so each row stays distinct and bounded

Why Does This Exist?

Self-attention is a weighted average of value vectors. Averages do not carry order. Shuffle three tokens and the same multiset of outputs returns, only reassigned to new slots. The sentence pair dog bit man versus man bit dog then scores identically, which is wrong.

Recurrence once gave order for free by reading left to right. Transformers dropped recurrence for parallel reads, and paid for speed with order blindness. Something must restore position before the first attention layer runs.

This page covers the additive fix: a per position vector added to each embedding. The rotation based successor lives in rotary position embeddings. The query key value machinery it feeds lives in self-attention.

Think of It Like This

Numbered stickers on moving boxes

A mover stacks plain brown boxes in a truck. By arrival they have slid around and no box shows where it stood. Now repeat the move with numbered stickers on every box: 0 on the first, 1 on the next, and so on. The truck still shakes, boxes still slide, but each box carries its slot number with it.

Token content is the box contents, position is the sticker number, attention mixing is the ride shaking boxes together, and adding the encoding is sticking the number on before loading. Reading sticker plus contents tells you what and where.

Where it stops: stickers never change what sits inside the box. Positional codes add no meaning to a word, they only let later layers recover which slot it came from.

How It Actually Works

Call $t$ the integer position, starting at 0. Call $d$ the model width, the embedding size. Call $i$ the dimension pair index, from $0$ to $d/2 - 1$ . The original construction assigns one sine and one cosine per pair:

PE_{(t, 2i)} = \sin\left(\frac{t}{10000^{2i/d}}\right), \qquad PE_{(t, 2i+1)} = \cos\left(\frac{t}{10000^{2i/d}}\right)

The denominator $10000^{2i/d}$ sets the wavelength for that pair. Small $i$ means a small denominator and a fast wave across $t$ . Large $i$ means a large denominator and a slow wave. The vector $PE_t$ is then added elementwise to the token embedding $x_t$ :

x_t' = x_t + PE_t

Attention only ever sees $x_t'$ , which carries content plus position in one vector.

Worked example: width 4, positions 0 and 1

Take $d = 4$ , so $i = 0$ and $i = 1$ . For $i = 0$ , the exponent is $0/4 = 0$ , so the denominator is $10000^0 = 1$ . For $i = 1$ , the exponent is $2/4 = 0.5$ , so the denominator is $10000^{0.5} = 100$ .

At $t = 0$ :

[\sin(0), \cos(0), \sin(0), \cos(0)] = [0, 1, 0, 1]

At $t = 1$ , the four angles are $1$ , $1$ , $0.01$ , and $0.01$ :

\sin(1) \approx 0.8415, \qquad \cos(1) \approx 0.5403

\sin(0.01) \approx 0.0100, \qquad \cos(0.01) \approx 1.0000

So position 1 encodes as $[0.84, 0.54, 0.01, 1.00]$ after rounding. Every entry stays inside $[-1, 1]$ no matter how large $t$ grows, and rows 0 and 1 differ in three slots. That is the diagram: two fast cells on the left carrying 0.84 and 0.54, two slow cells on the right carrying 0.01 and 1.00, with the row for position 0 reading $[0, 1, 0, 1]$ above it.

Why sine and cosine

Boundedness is the first win. Sine and cosine never leave $[-1, 1]$ , so late positions never shout louder than early ones. Relative structure is the second. For a fixed shift $k$ , $PE_{t+k}$ is a linear function of $PE_t$ , which gives the model a clean path to learn distance based patterns.

Code

Build the width 4 matrix and check boundedness plus distinct rows with plain Python.

1import math2
3
4def pe_row(t: int, d: int) -> list[float]:5    row: list[float] = []6    for i in range(d // 2):7        denom: float = 10000 ** ((2 * i) / d)8        angle: float = t / denom9        row.append(math.sin(angle))10        row.append(math.cos(angle))11    return row12
13
14rows: list[list[float]] = [pe_row(t, 4) for t in range(3)]15for r in rows:16    print([round(v, 2) for v in r])17# -> [0.0, 1.0, 0.0, 1.0]18# -> [0.84, 0.54, 0.01, 1.0]19# -> [0.91, -0.42, 0.02, 1.0]20
21flat: list[float] = [v for r in rows for v in r]22print(min(flat) >= -1.0 and max(flat) <= 1.0)  # -> True23print(rows[0] != rows[1])  # -> True

Position 1 prints $[0.84, 0.54, 0.01, 1.00]$ , matching the hand computation. All values sit in $[-1, 1]$ and no two rows match.

Watch Out For

Assembling attention with no position path

Symptom: a from scratch transformer trains without errors, shapes all match, yet word order shuffles barely change predictions and swapped clauses score the same. Fix: confirm a position vector is added before the first attention layer and that it actually reaches queries and keys, as in self-attention. If you moved to rotations, confirm the rotation is applied inside attention per rotary position embeddings.

Testing past the trained length and blaming the code

Symptom: quality falls once inputs run longer than anything seen in training, even though the sine formula still returns values in $[-1, 1]$ . Fix: treat trained length as a limit, not a guarantee. The model never shaped its weights around those late rows. Shorten, chunk, or switch to a length extension method instead of trusting raw extrapolation.

The Quick Version

Self-attention averages values, so order must be supplied from outside before the first layer.
Each position gets a sine and cosine vector with one frequency per dimension pair, added to the token embedding.
With width 4, position 0 is $[0, 1, 0, 1]$ and position 1 is $[0.84, 0.54, 0.01, 1.00]$ .
All entries stay in $[-1, 1]$ , so late positions never dominate early ones.
A fixed shift is a linear function of the current code, which helps distance patterns transfer.
Rotary embeddings replace addition with rotation but solve the same order blindness.