Long-Context Extension
Techniques that let a model trained on a shorter sequence generalize to much longer ones, by rescaling its positional signal rather than retraining it.
Position interpolation, NTK-aware scaling, and YaRN all work by rescaling a rotary position embedding model's rotation angles, mapping positions beyond the trained length back into the range the model actually learned to use well. This avoids retraining a model from scratch just to accept longer inputs, though most of these techniques still benefit from at least some fine-tuning at the target length.
They apply specifically to RoPE-based models — a model using ALiBi generalizes to longer sequences through its linear score penalty instead, with nothing to rescale.