Matrix Rank

Rank counts the linearly independent rows, which is always the same as the number of linearly independent columns. A $1000 \times 1000$ matrix whose every row is a multiple of the first has rank 1 — it looks large and contains one direction's worth of information.

That gap between size and rank is what low-rank methods exploit. A rank-$r$ matrix of shape $m \times n$ can be factored into an $m \times r$ and an $r \times n$ pair, which is a large saving when $r$ is small. This is the entire idea behind LoRA: instead of updating a full weight matrix, learn a low-rank correction to it and store a fraction of the parameters.

Rank is also a diagnostic. A feature matrix with rank below its column count means some features are exact combinations of others, and the closed-form regression solution stops being unique.