Linear Combination

A linear combination of $v_1, \ldots, v_k$ is $c_1 v_1 + c_2 v_2 + \cdots + c_k v_k$ for any choice of scalar coefficients. The set of everything you can reach this way is called the span of those vectors.

This is the operation neural networks are built from. A single neuron computes a linear combination of its inputs, weighted by its parameters, before applying a nonlinearity — the whole reason a nonlinearity is needed at all is that stacking linear combinations only ever produces another linear combination, so depth without activation buys nothing.

The idea also defines independence and therefore rank: a vector that is a linear combination of others adds no new direction. Attention is a linear combination too, where the coefficients are the attention weights and the vectors are the values.