Visual explainer
LoRA Low-Rank Tuning
A frozen weight matrix gains a trainable low-rank correction. Follow the factors, output sum, parameter count, initialization, merge and rank limit.
LoRA adapts a model by training small factor matrices while keeping selected base weights frozen. Full tuning with Adam can keep a gradient array g and two moment arrays m and v alongside W. These equal-size matrix silhouettes indicate values per parameter, not equal bytes: precision and optimizer implementation determine memory. Freezing W avoids its parameter gradients and optimizer state, but the base weights and activation computation remain.
Factor the correction
For a base matrix , LoRA learns and , with update . Here denotes scaling, commonly in the original LoRA method. Our synthetic example sets , , and . The drawn multiplication gives . Its second column is twice its first, so its rank is one. The restriction applies to the correction, not to W itself.
Count trainable values
A separate large-matrix example makes the count useful: full tuning changes values. Rank eight trains factor values. The ratio is exactly 256. Square areas encode counts at 256 weights per square drawing unit: the full square has side 256, while the aggregate factor square has side 16. Their areas differ by 256; the small square is not the actual tensor shape. This is a trainable-value ratio, not a total memory or speed ratio.
Run both paths
Return to the two-dimensional fixture and input . The base produces . The lower branch first computes , then . The sum is . The diagram's A-then-B execution order differs from the written matrix product BA because the rightmost matrix acts first.
Add the correction
The short white vector reaches ; the mint correction starts at its tip and ends at . A dashed vector connects the origin directly to the result. The coordinates use 30 drawing units per horizontal component and 35 per vertical component. The unequal axis scales are declared here; angles are not a measure of vector similarity. The output changes while W remains fixed. No accuracy comparison follows from this arithmetic.
Understand the rank budget
The left panel draws BA's columns and at 30 units per component: both lie on one line. The right panel draws independent unit columns at 130 units per component. This is a comparison of span, with an explicit scale change, not a comparison of magnitude. A correction BA has rank at most r and uses trainable values. Rank and the choice of adapted matrices require held-out evaluation; task labels such as style or reasoning do not determine a universally correct r.
A direction can disappear
Keep the learned factors but change the input to . A's weighted contributions are and , cancelling at zero. The number line uses 70 drawing units per scalar unit. Thus even though x is nonzero. The base still outputs . A different learned A could respond to this direction; this does not imply that LoRA cannot learn new facts.
Start with no correction
Rewind to initialization, before the learned B used earlier. Keep A nonzero, but set B to the zero column. Input still gives , then . The summed output remains the base's . A receives zero gradient through this branch initially, while B can receive a nonzero gradient. Initializing both factors to zero would instead block both factor gradients in this simple product.
Merge for serving
Restore the learned B. The merged weight is . One multiplication gives , matching the two branches. For a fixed linear adapter, merging removes its separate inference branch. This algebra assumes compatible arithmetic; quantized storage or rounding can alter numerical equivalence.
Where It Breaks
Change the desired correction to the identity matrix. It must send the two basis inputs to independent outputs and . Every rank-one correction has collinear columns, so none can match both exactly. The right panel shows one candidate, : it matches the first column and misses the second, with Frobenius error 1. Both panels use 130 units per component. This is a limit of a single linear correction, not proof that every network task needs a particular rank or that increasing rank always improves validation results.
The Quick Version
- Full tuning carries gradients and optimizer histories beside weights.
- LoRA factors the correction while freezing base weights.
- The large example uses 256 times fewer trainable values.
- The same input follows base and adapter paths.
- Adapter outputs add to the base output.
- Rank bounds the correction's independent output directions.
- A fixed adapter can ignore a nonzero input direction.
- Zero B preserves the initial base output.
- Merging reproduces the fixed linear adapter's output.
- Rank one cannot exactly match an identity correction.