Multicollinearity
When two input features move almost identically, a regression splits credit between them almost arbitrarily. Coefficients swing wildly, but predictions barely move.
Why Does This Exist?
An analyst predicts house prices from living area and room count, then reports what one extra room is worth. The fitted coefficient for rooms comes back negative, as if adding a room lowers the price. Refitting on a slightly different sample of the same streets flips it positive and changes its size severalfold. The price predictions themselves barely move between the two fits.
The fitting did nothing wrong. Living area and room count move together, and a linear regression coefficient answers a narrow question: what changes if this one feature moves while every other feature holds exactly still. When two features almost never move independently, that question has no stable answer from the data, even though their combined prediction stays rock solid.
This page covers why the coefficients wobble and how to measure it. The estimator underneath is on the ordinary least squares page, and the penalty that steadies the coefficients is on the ridge regression page.
The one idea you need first. A regression coefficient is credit assigned for unique variation. If a feature never varies without its partner varying too, there is no unique variation to assign, and the split between the two becomes a coin flip the optimizer happens to land on.
Think of It Like This
Two thermometers taped together
Tape two thermometers side by side on the same wall and ask which one deserves credit for telling you the room is warm. Every reading from one is matched by nearly the same reading from the other, so any split works: all the credit to the left one, all to the right one, half each. The room estimate from both together never changes, no matter how you divide it.
Sample a different week of readings and the winning split can swing wildly, because nothing in the data ever separated the two. That is multicollinearity: the pair is informative, the division is not.
Where the analogy stops: two taped thermometers measure the same thing by construction, while real features usually each carry something unique alongside the overlap. The overlap is what destabilizes the coefficients, and removing one feature can still discard genuine information.
How It Actually Works
Why the coefficients wobble while predictions hold
Write the prediction from two standardized features as , where and are the fitted coefficients. When and are nearly identical in every row, many pairs give almost the same sum: raising one while lowering the other leaves nearly unchanged. The optimizer faces a long flat valley of almost equally good pairs, and tiny noise decides where it stops. The sum, which drives every prediction, stays fixed while the individual weights slide along the valley.
Worked example
Take and correlated at , and a typical row with and . Two fits from different resamples land 45 points apart:
- Fit A uses and : .
- Fit B uses and : .
The coefficients disagree by 45 points on each weight, yet the predictions differ by 0.9 on a value near 101, under one percent. Same story as the house prices: wild coefficients, calm predictions.
Measuring it with VIF
The variance inflation factor turns the overlap into a number. Regress feature on all the other features, take that regression's , written , and compute:
near 0 means the feature stands alone and VIF sits near 1, no inflation. near 1 means the other features almost fully predict it, and VIF grows without bound. Values above roughly 5 to 10 are the commonly cited range for flagging a real problem. Here gives and , far past any threshold. For pure forecasting this often matters little, since predictions hold steady. It matters the moment anyone interprets one coefficient on its own.
Code
The point of this snippet is the check: turn a correlation into a VIF before trusting any individual coefficient.
def vif_from_r_squared(r_squared: float) -> float: return 1.0 / (1.0 - r_squared)
r: float = 0.99r2: float = r ** 2print(round(r2, 4))# -> 0.9801print(round(vif_from_r_squared(r2), 1))# -> 50.3A VIF of 50.3 means this coefficient's variance is inflated fiftyfold by overlap with the other features. Read the prediction, not the weight.
Watch Out For
Deleting a feature because its coefficient looks wrong
The symptom is a negative sign on a feature you know helps, or a weight that flips every time the sample changes, followed by the urge to drop the feature entirely. But a strange coefficient under collinearity indicts the split, not the feature. Removing a genuinely informative input discards real signal and can make predictions worse.
Check the correlation matrix and VIF first. If the partner feature carries the same information, either keep both and stop interpreting single weights, or steady them with a penalty, as the ridge regression page shows.
Believing a p-value on a collinear coefficient
The symptom is an important feature testing as statistically insignificant, so it gets cut from the report. Collinearity inflates that coefficient's standard error, which shrinks its test statistic mechanically. The p-value is reporting the overlap, not the feature's worth.
Judge the group before the individual: a joint test of the collinear set stays valid while single-coefficient tests wobble. Never drop a feature on one collinear p-value alone.
The Quick Version
- Multicollinearity is overlap between input features, and it lets many different coefficient splits produce nearly identical predictions.
- Individual coefficients swing wildly across resamples, even flipping sign, while the model's predictions stay stable.
- VIF scores the overlap directly: gives and a VIF of 50.3, far past the flag range of 5 to 10.
- It barely matters for pure forecasting and matters enormously for interpreting any single coefficient.
- Check VIF before dropping a feature or believing one collinear p-value.