Matrix Factorization
Rebuild a mostly empty ratings grid as the product of two small tables. Fit only on cells you saw and every blank cell inherits a prediction for free.
Why Does This Exist?
A shop has 40,000 users and 30,000 items, and each user rated a few dozen items at most. The ratings grid is huge and almost entirely blank, yet the business question is about the blanks: what would this user rate that untouched item. Averaging what little exists per user or per item collapses on sparse rows.
Principal component analysis and the singular value decomposition it uses both need every cell filled before they start. Neither runs on a grid with holes. Matrix factorization is the version of split one big grid into two small ones built for holes: fit only on seen cells, then read the product everywhere, including cells never seen. This page covers that split, the loss that ignores blanks, and the two solvers. The shopping use case lives in collaborative filtering. The newcomer problem lives in cold start problem.
One basic makes the rest click. A dot product multiplies two short rows entry by entry and adds the results, so dotted with is . A rank is just how many numbers describe each user and each item.
Think of It Like This
Guessing taste from a few tried dishes
A food app knows a scattered handful of dishes each customer rated out of thousands on the menu. Nobody tried more than a sliver. You want to guess how one customer rates a dish never ordered.
Describe every customer by a short taste list, such as spice liking and sweetness liking, and every dish by how much of each quality it holds. The predicted rating is how well the two lists line up. You never needed that customer to try that dish. You only needed enough other ratings to pin down the taste list, and then every untried dish gets a number for free.
Where it stops: real taste shifts with mood, company and season, while the two short lists sit still. The model also cannot describe a newcomer with zero ratings, because no tried dish exists to pin the list down.
How It Actually Works
Call the ratings grid with users by items. Call the user table with shape by and the item table with shape by , where is far smaller than or . Row is user taste list, row is item quality list, and the prediction for any pair is . That dot product exists for every pair, seen or not, which is why blanks inherit values without ever being fit directly.
Fit only on seen cells with squared error plus a penalty that keeps numbers small:
Here is the regularization strength, sums squared entries of , and unseen cells contribute zero to the sum and send zero gradient. Two solvers dominate. Gradient descent updates both tables together. Alternating least squares fixes , solves for exactly, then fixes and solves for , repeating until stable. The second turns one hard joint problem into a run of easy ones.
Worked example: three seen cells pin two short lists
Take . A user row is and an item row is . The prediction is:
Now score three held-out cells. True ratings are and the product predicts . Errors are . Square them:
Add and average:
Root it for RMSE:
Three numbers pin the lists, and the same lists price unseen pairs to within about a quarter star. On a fully seen grid with no penalty this converges toward the same low-rank shape the SVD gives directly, trading exact optimality for the ability to run on grids the SVD cannot touch.
Code
The point is the check: rebuild the dot product and the RMSE from the worked example.
import math
u: list[float] = [0.9, 0.1]v: list[float] = [1.0, 0.5]pred: float = u[0] * v[0] + u[1] * v[1]print(round(pred, 2)) # -> 0.95
true: list[float] = [4.0, 3.0, 2.0]guess: list[float] = [3.7, 3.2, 1.8]sq: list[float] = [(t - g) ** 2 for t, g in zip(true, guess)]rmse: float = math.sqrt(sum(sq) / len(sq))print([round(x, 2) for x in sq]) # -> [0.09, 0.04, 0.04]print(round(rmse, 2)) # -> 0.24The printout matches the hand computation: dot product 0.95, squared errors 0.09, 0.04 and 0.04, RMSE about 0.24. Change one guess by 0.5 and watch RMSE jump, which shows how directly blanks inherit the two short lists.
Watch Out For
Picking rank by fit on seen cells alone
Raising always lowers error on cells used for fitting, the same way extra splits help random forests memorize training rows. Past a point the tables memorize specific ratings instead of taste structure.
Symptom: training error keeps falling while held-out error rises. Fix: hold out some seen cells before fitting, pick and on those held-out cells only, and watch overfitting underfitting signals during the run.
Trusting predictions for newcomers with no history
A user with zero seen ratings has a row in that never received gradient, so the product outputs a number that reflects initialization, not taste. The same holds for a new item with an unfitted row in .
Symptom: confident-looking scores for brand-new users or items that later prove random. Fix: route newcomers through popularity or content rules from cold start problem until enough ratings exist to fit their rows.
The Quick Version
- Matrix factorization writes an by sparse grid as user table by times item table by with small .
- The loss sums squared error only on seen cells plus a penalty, so blanks send zero gradient.
- User list 0.90 and 0.10 dotted with item list 1.00 and 0.50 gives 0.95.
- True ratings 4.00, 3.00 and 2.00 against guesses 3.70, 3.20 and 1.80 give squared errors 0.09, 0.04 and 0.04 with RMSE about 0.24.
- Alternating least squares fixes one table and solves the other exactly, repeating until stable.
- New users or items with no seen cells have unfitted rows, so their scores need a fallback rule.