ALiBi Attention Bias
A positional scheme that adds no vector to the input at all, instead subtracting a distance-proportional penalty from attention scores before softmax runs.
Think of It Like This
Like a conversation in a noisy hall where you naturally pay less attention to voices the further away they stand, without needing to tag everyone with a name badge.
Short for Attention with Linear Biases, ALiBi computes the ordinary score and then subtracts from it, where is a fixed, unlearned slope specific to each attention head and is the distance between the query and key positions. No positional embedding touches the query, key, or value vectors at any point.
Because the penalty is the same linear function of distance at any sequence length, ALiBi generalizes to sequences longer than its training length noticeably better than additive or rotary schemes, without needing a separate rescaling step.