Skip to content
AI360XpertBeta

Long-Context Extension

Long-context extension rescales a model's positional signal so sequences longer than its trained length still land back in the range it actually learned.

Position 12 of 16 is read as position 6 after scaling by one half, so its RoPE angle stays inside the trained 0 to 8 range
Position 12 of 16 is read as position 6 after scaling by one half, so its RoPE angle stays inside the trained 0 to 8 range

Why Does This Exist?

Every model ships with a fixed context window: the run of tokens it can read in one pass. You can't prompt your way past it.

The reason here is positional. Rotary position embeddings rotate each query and key by an angle tied to position, and your model only learned angles up to its trained length. Feed it positions past that edge and the formula still returns a number, but attention never shaped itself around angles that large. Quality slides, often right past the edge.

Retraining from zero at a longer length would fix it. It would also charge a full pretraining run. This page covers the cheaper family instead: keep the weights, remap positions back into familiar territory, then touch up with a short fine-tune. The frame itself lives on the context-windows page. The rotation trick it remaps lives on the RoPE page. Here you get the remap.

The one idea you need first: RoPE turns distance into angle. A query at position mm and a key at position nn meet at an angle set by m−nm - n, never by mm or nn alone. Here mm is the query position, nn is the key position, and m−nm - n is how far apart they sit. If you can keep every rescaled angle inside the range training saw, relative distances keep meaning something.

Think of It Like This

A short ruler and a photocopier set to 50 percent

You own a ruler marked 0 to 8. Now someone hands you a 16-inch shelf to measure. You could guess past the end. Better: photocopy the shelf plan at 50 percent, then measure the copy. The 12-inch mark on the shelf lands on the 6-inch mark of your copy, and the far end lands exactly on 8. You read only marks your ruler already knows.

That's position interpolation in miniature. The ruler is your trained length. The shelf is the longer input. The 50 percent copy is the scale s=L/L′s = L / L', where LL is the length you trained on, L′L' is the length you want, and ss is the shrink factor. The reading step never touches unfamiliar marks.

Where the ruler stops: ruler marks sit still side by side, but RoPE frequencies interact. Shrinking every frequency by the same half blurs the fast ones that tell nearby tokens apart. That blur is why uniform shrinking is only the start, and why the next section treats frequencies differently.

How It Actually Works

Three ideas stack. One formula folds positions back. One fix protects local detail. One refinement steadies the softmax. Then numbers.

The one formula

Call LL the trained length, the longest run your model saw in training. Call L′L' the target length, the longer run you want now. Call pp any raw position from 00 to L′L', and call ss the scale s=L/L′s = L / L', a number below 1 when you extend. The remap is one multiplication:

pscaled=p×sp_{scaled} = p \times s

Here pscaledp_{scaled} is the position you actually feed to RoPE, the familiar-territory stand-in for pp. For one frequency pair, call θ\theta its rotation rate, the radians per position step for that pair. The angle attention sees is:

angle=pscaled×θangle = p_{scaled} \times \theta

So position L′L' lands exactly on LL, because L′×(L/L′)=LL' \times (L / L') = L. Everything longer folds evenly back inside.

Why one scale blurs nearby tokens

RoPE does not use one θ\theta. It uses many, one per 2D pair. Fast pairs with large θ\theta spin quickly and tell apart tokens one or two steps away. Slow pairs with small θ\theta spin slowly and carry long-range order. Here fast means large θ\theta, slow means small θ\theta.

Position interpolation squeezes all of them by the same ss. That keeps the math simple and it does pull distant positions back inside. But it also squeezes the fast pairs that were already fine, dulling the very signal that separates neighbours. NTK-aware scaling answers that: leave high frequencies near untouched, push most of the squeeze onto low frequencies where extrapolation actually breaks. You keep local sharpness and still fix the far edge.

YaRN: ramp plus temperature

YaRN blends the two extremes instead of picking one. Each frequency gets its own factor from a smooth ramp: near 1 (almost untouched) for the fastest pairs, near ss (fully squeezed) for the slowest, sliding between. Because a longer run spreads attention mass thinner, YaRN also nudges the softmax temperature to counter the flattening. In practice you still run a short fine-tune at L′L' after rescaling. It is shorter than pretraining, but it is not zero.

Worked example: 8 becomes 16

Take L=8L = 8 and L′=16L' = 16, so s=8/16=0.5s = 8 / 16 = 0.5. Fix one frequency at θ=0.1\theta = 0.1 radians per step to keep the arithmetic visible. Here θ\theta is just that pair's rate, and all angles below are in radians.

Pick p=12p = 12, a position your 8-long training never saw:

pscaled=12×0.5=6,angle=6×0.1=0.6p_{scaled} = 12 \times 0.5 = 6, \quad angle = 6 \times 0.1 = 0.6

The far edge lands exactly home:

p=16→pscaled=8,angle=8×0.1=0.8p = 16 \rightarrow p_{scaled} = 8, \quad angle = 8 \times 0.1 = 0.8

That 0.8 is the largest angle this frequency ever saw in training. Without the remap, p=12p = 12 would have asked for 12×0.1=1.212 \times 0.1 = 1.2, well past 0.8 and outside familiar territory. The diagram above draws this same fold: 12 sits out on the long bar and lands on 6, while 16 lands on 8.

The trade is honest. You buy length without pretraining, and you pay in two places: local sharpness softens under uniform squeeze, and compute still grows fast while the KV cache grows with every extra token. Don't use this when a few passages would do: if only a handful of chunks matter, retrieve them instead of widening the whole frame. It assumes rescaled relative distances still signal order. When that fails, middle-context recall goes uneven first, and a longer accepted window hides it.

Code

The point here is the check, not the formula retyped. Rescale first, then turn positions into angles, and confirm the edge lands home.

def rescaled_angle(position: int, theta: float, trained_len: int, target_len: int) -> float:    scale = trained_len / target_len  # s, below 1 when extending    return position * scale * theta

theta = 0.1trained_len, target_len = 8, 16  # L and L prime from the worked example
print(rescaled_angle(12, theta, trained_len, target_len))  # -> 0.6print(rescaled_angle(16, theta, trained_len, target_len))  # -> 0.8print(12 * theta)  # -> 1.2

Position 12 reads as 0.6 instead of 1.2, and position 16 lands on 0.8, the trained edge from the example above. Same numbers as the prose and the picture, one thread.

Watch Out For

Accepting more tokens is not the same as using them

Your window now accepts 16 tokens. Your recall is not automatically uniform across all 16. The symptom is specific: edge questions pass while middle questions miss, so a demo looks fine and a real document task drops facts from the center.

Fix it by probing the middle at the target length before you claim the win. Ask for facts planted early, middle, and late in a long input. If the middle sags, say so, and reach for retrieval or attention sinks and streaming where they fit instead of widening further.

Rescaling the wrong position scheme, or skipping the touch-up

You apply interpolation to a model that never used RoPE, or you apply it and ship zero-shot. The symptom differs by mistake. On an ALiBi-style model there is no angle to rescale, so nothing improves and you have patched the wrong mechanism. On a RoPE model with no fine-tune, neighbours blur: order flips on close tokens and off-by-one errors creep in where local detail lived.

Fix it in order. First confirm the position scheme: only RoPE angles respond to this remap. Then run the short fine-tune at L′L' and re-eval at L′L', not at LL. If you can't afford that touch-up and its eval, don't claim the longer length.

The Quick Version

  • Long-context extension remaps positions past the trained length back into 0 to L, so RoPE angles stay inside familiar territory without retraining from zero.
  • Position interpolation multiplies each raw position by L over L prime. With L of 8 and L prime of 16, position 12 is read as position 6.
  • NTK-aware scaling leaves fast high-frequency pairs near untouched and concentrates the squeeze on slow low-frequency pairs that handle long range.
  • YaRN ramps per-frequency scales between untouched and fully squeezed and steadies the flattened softmax with a temperature tweak, usually plus a short fine-tune at the target length.
  • A longer accepted window still charges quadratic attention work and linear KV cache memory, and extension suits RoPE models only. Few-passage tasks favor retrieval over widening.