Skip to content
AI360XpertBeta

Multicollinearity

When two input features move almost identically, a regression splits credit between them almost arbitrarily. Coefficients swing wildly, but predictions barely move.

Two fits split credit anywhere along the line w1 plus w2 equals 100 yet both predict about 101, so coefficients swing 45 points while predictions differ under one percent
Two fits split credit anywhere along the line w1 plus w2 equals 100 yet both predict about 101, so coefficients swing 45 points while predictions differ under one percent

Why Does This Exist?

An analyst predicts house prices from living area and room count, then reports what one extra room is worth. The fitted coefficient for rooms comes back negative, as if adding a room lowers the price. Refitting on a slightly different sample of the same streets flips it positive and changes its size severalfold. The price predictions themselves barely move between the two fits.

The fitting did nothing wrong. Living area and room count move together, and a linear regression coefficient answers a narrow question: what changes if this one feature moves while every other feature holds exactly still. When two features almost never move independently, that question has no stable answer from the data, even though their combined prediction stays rock solid.

This page covers why the coefficients wobble and how to measure it. The estimator underneath is on the ordinary least squares page, and the penalty that steadies the coefficients is on the ridge regression page.

The one idea you need first. A regression coefficient is credit assigned for unique variation. If a feature never varies without its partner varying too, there is no unique variation to assign, and the split between the two becomes a coin flip the optimizer happens to land on.

Think of It Like This

Two thermometers taped together

Tape two thermometers side by side on the same wall and ask which one deserves credit for telling you the room is warm. Every reading from one is matched by nearly the same reading from the other, so any split works: all the credit to the left one, all to the right one, half each. The room estimate from both together never changes, no matter how you divide it.

Sample a different week of readings and the winning split can swing wildly, because nothing in the data ever separated the two. That is multicollinearity: the pair is informative, the division is not.

Where the analogy stops: two taped thermometers measure the same thing by construction, while real features usually each carry something unique alongside the overlap. The overlap is what destabilizes the coefficients, and removing one feature can still discard genuine information.

How It Actually Works

Why the coefficients wobble while predictions hold

Write the prediction from two standardized features as w1z1+w2z2w_1 z_1 + w_2 z_2, where w1w_1 and w2w_2 are the fitted coefficients. When z1z_1 and z2z_2 are nearly identical in every row, many pairs (w1,w2)(w_1, w_2) give almost the same sum: raising one while lowering the other leaves w1z1+w2z2w_1 z_1 + w_2 z_2 nearly unchanged. The optimizer faces a long flat valley of almost equally good pairs, and tiny noise decides where it stops. The sum, which drives every prediction, stays fixed while the individual weights slide along the valley.

Worked example

Take z1z_1 and z2z_2 correlated at r=0.99r = 0.99, and a typical row with z1=1.0z_1 = 1.0 and z2=1.02z_2 = 1.02. Two fits from different resamples land 45 points apart:

  • Fit A uses w1=30w_1 = 30 and w2=70w_2 = 70: 30×1.0+70×1.02=30+71.4=101.430 \times 1.0 + 70 \times 1.02 = 30 + 71.4 = 101.4.
  • Fit B uses w1=75w_1 = 75 and w2=25w_2 = 25: 75×1.0+25×1.02=75+25.5=100.575 \times 1.0 + 25 \times 1.02 = 75 + 25.5 = 100.5.

The coefficients disagree by 45 points on each weight, yet the predictions differ by 0.9 on a value near 101, under one percent. Same story as the house prices: wild coefficients, calm predictions.

Measuring it with VIF

The variance inflation factor turns the overlap into a number. Regress feature ii on all the other features, take that regression's R2R^2, written Ri2R_i^2, and compute:

VIFi=11−Ri2\text{VIF}_i = \frac{1}{1 - R_i^2}

Ri2R_i^2 near 0 means the feature stands alone and VIF sits near 1, no inflation. Ri2R_i^2 near 1 means the other features almost fully predict it, and VIF grows without bound. Values above roughly 5 to 10 are the commonly cited range for flagging a real problem. Here r=0.99r = 0.99 gives R2=0.9801R^2 = 0.9801 and VIF=1/0.0199=50.3\text{VIF} = 1 / 0.0199 = 50.3, far past any threshold. For pure forecasting this often matters little, since predictions hold steady. It matters the moment anyone interprets one coefficient on its own.

Code

The point of this snippet is the check: turn a correlation into a VIF before trusting any individual coefficient.

def vif_from_r_squared(r_squared: float) -> float:    return 1.0 / (1.0 - r_squared)

r: float = 0.99r2: float = r ** 2print(round(r2, 4))# -> 0.9801print(round(vif_from_r_squared(r2), 1))# -> 50.3

A VIF of 50.3 means this coefficient's variance is inflated fiftyfold by overlap with the other features. Read the prediction, not the weight.

Watch Out For

Deleting a feature because its coefficient looks wrong

The symptom is a negative sign on a feature you know helps, or a weight that flips every time the sample changes, followed by the urge to drop the feature entirely. But a strange coefficient under collinearity indicts the split, not the feature. Removing a genuinely informative input discards real signal and can make predictions worse.

Check the correlation matrix and VIF first. If the partner feature carries the same information, either keep both and stop interpreting single weights, or steady them with a penalty, as the ridge regression page shows.

Believing a p-value on a collinear coefficient

The symptom is an important feature testing as statistically insignificant, so it gets cut from the report. Collinearity inflates that coefficient's standard error, which shrinks its test statistic mechanically. The p-value is reporting the overlap, not the feature's worth.

Judge the group before the individual: a joint test of the collinear set stays valid while single-coefficient tests wobble. Never drop a feature on one collinear p-value alone.

The Quick Version

  • Multicollinearity is overlap between input features, and it lets many different coefficient splits produce nearly identical predictions.
  • Individual coefficients swing wildly across resamples, even flipping sign, while the model's predictions stay stable.
  • VIF scores the overlap directly: r=0.99r = 0.99 gives R2=0.9801R^2 = 0.9801 and a VIF of 50.3, far past the flag range of 5 to 10.
  • It barely matters for pure forecasting and matters enormously for interpreting any single coefficient.
  • Check VIF before dropping a feature or believing one collinear p-value.