Self-Attention
A mechanism in transformers that allows every token in a sequence to dynamically weigh the importance of every other token to understand full context.
Think of It Like This
Like reading a mystery novel where your brain automatically connects a clue on page 10 directly to the villain revealed on page 300.
Unlike recurrent networks that process sequences linearly, self-attention computes relationships globally across the entire context window in parallel. The word 'bank' pays attention to 'river' or 'money' elsewhere in the sentence to determine its own meaning. This massive parallelization and contextual depth revolutionized NLP.