Long-Context Extension
Long-context extension rescales a model's positional signal so sequences longer than its trained length still land back in the range it actually learned.
Why Does This Exist?
Every model ships with a fixed context window: the run of tokens it can read in one pass. You can't prompt your way past it.
The reason here is positional. Rotary position embeddings rotate each query and key by an angle tied to position, and your model only learned angles up to its trained length. Feed it positions past that edge and the formula still returns a number, but attention never shaped itself around angles that large. Quality slides, often right past the edge.
Retraining from zero at a longer length would fix it. It would also charge a full pretraining run. This page covers the cheaper family instead: keep the weights, remap positions back into familiar territory, then touch up with a short fine-tune. The frame itself lives on the context-windows page. The rotation trick it remaps lives on the RoPE page. Here you get the remap.
The one idea you need first: RoPE turns distance into angle. A query at position and a key at position meet at an angle set by , never by or alone. Here is the query position, is the key position, and is how far apart they sit. If you can keep every rescaled angle inside the range training saw, relative distances keep meaning something.
Think of It Like This
A short ruler and a photocopier set to 50 percent
You own a ruler marked 0 to 8. Now someone hands you a 16-inch shelf to measure. You could guess past the end. Better: photocopy the shelf plan at 50 percent, then measure the copy. The 12-inch mark on the shelf lands on the 6-inch mark of your copy, and the far end lands exactly on 8. You read only marks your ruler already knows.
That's position interpolation in miniature. The ruler is your trained length. The shelf is the longer input. The 50 percent copy is the scale , where is the length you trained on, is the length you want, and is the shrink factor. The reading step never touches unfamiliar marks.
Where the ruler stops: ruler marks sit still side by side, but RoPE frequencies interact. Shrinking every frequency by the same half blurs the fast ones that tell nearby tokens apart. That blur is why uniform shrinking is only the start, and why the next section treats frequencies differently.
How It Actually Works
Three ideas stack. One formula folds positions back. One fix protects local detail. One refinement steadies the softmax. Then numbers.
The one formula
Call the trained length, the longest run your model saw in training. Call the target length, the longer run you want now. Call any raw position from to , and call the scale , a number below 1 when you extend. The remap is one multiplication:
Here is the position you actually feed to RoPE, the familiar-territory stand-in for . For one frequency pair, call its rotation rate, the radians per position step for that pair. The angle attention sees is:
So position lands exactly on , because . Everything longer folds evenly back inside.
Why one scale blurs nearby tokens
RoPE does not use one . It uses many, one per 2D pair. Fast pairs with large spin quickly and tell apart tokens one or two steps away. Slow pairs with small spin slowly and carry long-range order. Here fast means large , slow means small .
Position interpolation squeezes all of them by the same . That keeps the math simple and it does pull distant positions back inside. But it also squeezes the fast pairs that were already fine, dulling the very signal that separates neighbours. NTK-aware scaling answers that: leave high frequencies near untouched, push most of the squeeze onto low frequencies where extrapolation actually breaks. You keep local sharpness and still fix the far edge.
YaRN: ramp plus temperature
YaRN blends the two extremes instead of picking one. Each frequency gets its own factor from a smooth ramp: near 1 (almost untouched) for the fastest pairs, near (fully squeezed) for the slowest, sliding between. Because a longer run spreads attention mass thinner, YaRN also nudges the softmax temperature to counter the flattening. In practice you still run a short fine-tune at after rescaling. It is shorter than pretraining, but it is not zero.
Worked example: 8 becomes 16
Take and , so . Fix one frequency at radians per step to keep the arithmetic visible. Here is just that pair's rate, and all angles below are in radians.
Pick , a position your 8-long training never saw:
The far edge lands exactly home:
That 0.8 is the largest angle this frequency ever saw in training. Without the remap, would have asked for , well past 0.8 and outside familiar territory. The diagram above draws this same fold: 12 sits out on the long bar and lands on 6, while 16 lands on 8.
The trade is honest. You buy length without pretraining, and you pay in two places: local sharpness softens under uniform squeeze, and compute still grows fast while the KV cache grows with every extra token. Don't use this when a few passages would do: if only a handful of chunks matter, retrieve them instead of widening the whole frame. It assumes rescaled relative distances still signal order. When that fails, middle-context recall goes uneven first, and a longer accepted window hides it.
Code
The point here is the check, not the formula retyped. Rescale first, then turn positions into angles, and confirm the edge lands home.
def rescaled_angle(position: int, theta: float, trained_len: int, target_len: int) -> float: scale = trained_len / target_len # s, below 1 when extending return position * scale * theta
theta = 0.1trained_len, target_len = 8, 16 # L and L prime from the worked example
print(rescaled_angle(12, theta, trained_len, target_len)) # -> 0.6print(rescaled_angle(16, theta, trained_len, target_len)) # -> 0.8print(12 * theta) # -> 1.2Position 12 reads as 0.6 instead of 1.2, and position 16 lands on 0.8, the trained edge from the example above. Same numbers as the prose and the picture, one thread.
Watch Out For
Accepting more tokens is not the same as using them
Your window now accepts 16 tokens. Your recall is not automatically uniform across all 16. The symptom is specific: edge questions pass while middle questions miss, so a demo looks fine and a real document task drops facts from the center.
Fix it by probing the middle at the target length before you claim the win. Ask for facts planted early, middle, and late in a long input. If the middle sags, say so, and reach for retrieval or attention sinks and streaming where they fit instead of widening further.
Rescaling the wrong position scheme, or skipping the touch-up
You apply interpolation to a model that never used RoPE, or you apply it and ship zero-shot. The symptom differs by mistake. On an ALiBi-style model there is no angle to rescale, so nothing improves and you have patched the wrong mechanism. On a RoPE model with no fine-tune, neighbours blur: order flips on close tokens and off-by-one errors creep in where local detail lived.
Fix it in order. First confirm the position scheme: only RoPE angles respond to this remap. Then run the short fine-tune at and re-eval at , not at . If you can't afford that touch-up and its eval, don't claim the longer length.
The Quick Version
- Long-context extension remaps positions past the trained length back into 0 to L, so RoPE angles stay inside familiar territory without retraining from zero.
- Position interpolation multiplies each raw position by L over L prime. With L of 8 and L prime of 16, position 12 is read as position 6.
- NTK-aware scaling leaves fast high-frequency pairs near untouched and concentrates the squeeze on slow low-frequency pairs that handle long range.
- YaRN ramps per-frequency scales between untouched and fully squeezed and steadies the flattened softmax with a temperature tweak, usually plus a short fine-tune at the target length.
- A longer accepted window still charges quadratic attention work and linear KV cache memory, and extension suits RoPE models only. Few-passage tasks favor retrieval over widening.