Skip to content
AI360XpertBeta
Core ML

Visual explainer

LoRA Low-Rank Tuning

A frozen weight matrix gains a trainable low-rank correction. Follow the factors, output sum, parameter count, initialization, merge and rank limit.

Full tuning keeps gradient and optimizer arrays alongside the weight matrix.
Full tuning keeps gradient and optimizer arrays alongside the weight matrix.

LoRA adapts a model by training small factor matrices while keeping selected base weights frozen. Full tuning with Adam can keep a gradient array g and two moment arrays m and v alongside W. These equal-size matrix silhouettes indicate values per parameter, not equal bytes: precision and optimizer implementation determine memory. Freezing W avoids its parameter gradients and optimizer state, but the base weights and activation computation remain.

Factor the correction

A column times a row produces a rank-one weight update.
A column times a row produces a rank-one weight update.

For a base matrix W∈Rd×kW\in\mathbb R^{d\times k}, LoRA learns A∈Rr×kA\in\mathbb R^{r\times k} and B∈Rd×rB\in\mathbb R^{d\times r}, with update ΔW=sBA\Delta W=sBA. Here ss denotes scaling, commonly α/r\alpha/r in the original LoRA method. Our synthetic example sets s=1s=1, W=I2W=I_2, A=(1,2)A=(1,2) and B=(3,1)TB=(3,1)^T. The drawn multiplication gives BA=(3612)BA=\begin{pmatrix}3&6\\1&2\end{pmatrix}. Its second column is twice its first, so its rank is one. The restriction applies to the correction, not to W itself.

Count trainable values

Rank eight uses 256 times fewer trainable values for a 4096-square matrix.
Rank eight uses 256 times fewer trainable values for a 4096-square matrix.

A separate large-matrix example makes the count useful: full tuning changes 40962=16,777,2164096^2=16,777,216 values. Rank eight trains 8(4096+4096)=65,5368(4096+4096)=65,536 factor values. The ratio is exactly 256. Square areas encode counts at 256 weights per square drawing unit: the full square has side 256, while the aggregate factor square has side 16. Their areas differ by 256; the small square is not the actual tensor shape. This is a trainable-value ratio, not a total memory or speed ratio.

Run both paths

The same input follows a frozen path and a learned low-rank path before addition.
The same input follows a frozen path and a learned low-rank path before addition.

Return to the two-dimensional fixture and input x=(2,1)Tx=(2,1)^T. The base produces Wx=(2,1)TWx=(2,1)^T. The lower branch first computes Ax=1(2)+2(1)=4Ax=1(2)+2(1)=4, then B(Ax)=(12,4)TB(Ax)=(12,4)^T. The sum is h=Wx+BAx=(14,5)Th=Wx+BAx=(14,5)^T. The diagram's A-then-B execution order differs from the written matrix product BA because the rightmost matrix acts first.

Add the correction

The learned correction adds to the base output rather than replacing it.
The learned correction adds to the base output rather than replacing it.

The short white vector reaches (2,1)(2,1); the mint correction (12,4)(12,4) starts at its tip and ends at (14,5)(14,5). A dashed vector connects the origin directly to the result. The coordinates use 30 drawing units per horizontal component and 35 per vertical component. The unequal axis scales are declared here; angles are not a measure of vector similarity. The output changes while W remains fixed. No accuracy comparison follows from this arithmetic.

Understand the rank budget

Rank limits how many independent output directions one update can span.
Rank limits how many independent output directions one update can span.

The left panel draws BA's columns (3,1)(3,1) and (6,2)(6,2) at 30 units per component: both lie on one line. The right panel draws independent unit columns at 130 units per component. This is a comparison of span, with an explicit scale change, not a comparison of magnitude. A correction BA has rank at most r and uses r(d+k)r(d+k) trainable values. Rank and the choice of adapted matrices require held-out evaluation; task labels such as style or reasoning do not determine a universally correct r.

A direction can disappear

A rank-one branch can give zero correction to a nonzero input.
A rank-one branch can give zero correction to a nonzero input.

Keep the learned factors but change the input to x=(−2,1)Tx=(-2,1)^T. A's weighted contributions are 1(−2)=−21(-2)=-2 and 2(1)=+22(1)=+2, cancelling at zero. The number line uses 70 drawing units per scalar unit. Thus BAx=B(0)=0BAx=B(0)=0 even though x is nonzero. The base still outputs Wx=(−2,1)TWx=(-2,1)^T. A different learned A could respond to this direction; this does not imply that LoRA cannot learn new facts.

Start with no correction

Initializing B to zero preserves the base output at the start of training.
Initializing B to zero preserves the base output at the start of training.

Rewind to initialization, before the learned B used earlier. Keep A nonzero, but set B to the zero column. Input (2,1)(2,1) still gives Ax=4Ax=4, then B(Ax)=0B(Ax)=0. The summed output remains the base's (2,1)(2,1). A receives zero gradient through this branch initially, while B can receive a nonzero gradient. Initializing both factors to zero would instead block both factor gradients in this simple product.

Merge for serving

Merging the correction into the base produces the same linear output.
Merging the correction into the base produces the same linear output.

Restore the learned B. The merged weight is W′=W+BA=(4613)W'=W+BA=\begin{pmatrix}4&6\\1&3\end{pmatrix}. One multiplication gives W′x=(4(2)+6(1),1(2)+3(1))T=(14,5)TW'x=(4(2)+6(1),1(2)+3(1))^T=(14,5)^T, matching the two branches. For a fixed linear adapter, merging removes its separate inference branch. This algebra assumes compatible arithmetic; quantized storage or rounding can alter numerical equivalence.

Where It Breaks

A rank-one correction cannot reproduce two independent target columns.
A rank-one correction cannot reproduce two independent target columns.

Change the desired correction to the identity matrix. It must send the two basis inputs to independent outputs (1,0)(1,0) and (0,1)(0,1). Every rank-one correction has collinear columns, so none can match both exactly. The right panel shows one candidate, diag(1,0)\mathrm{diag}(1,0): it matches the first column and misses the second, with Frobenius error 1. Both panels use 130 units per component. This is a limit of a single linear correction, not proof that every network task needs a particular rank or that increasing rank always improves validation results.

The Quick Version

  • Full tuning carries gradients and optimizer histories beside weights.
  • LoRA factors the correction while freezing base weights.
  • The large example uses 256 times fewer trainable values.
  • The same input follows base and adapter paths.
  • Adapter outputs add to the base output.
  • Rank bounds the correction's independent output directions.
  • A fixed adapter can ignore a nonzero input direction.
  • Zero B preserves the initial base output.
  • Merging reproduces the fixed linear adapter's output.
  • Rank one cannot exactly match an identity correction.