Skip to content
AI360XpertBeta
Glossary
Definition

Matrix Rank

The number of genuinely independent directions in a matrix — how much distinct information it actually carries.

Think of It Like This

Like counting how many musicians in a brass band are playing distinct melodies rather than duplicating each other's sheet music in unison.

Rank counts the linearly independent rows, which is always the same as the number of linearly independent columns. A 1000×10001000 \times 1000 matrix whose every row is a multiple of the first has rank 1 — it looks large and contains one direction's worth of information.

That gap between size and rank is what low-rank methods exploit. A rank-rr matrix of shape m×nm \times n can be factored into an m×rm \times r and an r×nr \times n pair, which is a large saving when rr is small. This is the entire idea behind LoRA: instead of updating a full weight matrix, learn a low-rank correction to it and store a fraction of the parameters.

Rank is also a diagnostic. A feature matrix with rank below its column count means some features are exact combinations of others, and the closed-form regression solution stops being unique.