2

I have read that when using Ordinary Least Squares (OLS) for multiple linear regression, the coefficients/weights are unreliable for predictor variables that are collinear. I was wondering if this is also the case for regularisation methods (ridge/lasso/elastic net regression) when variables are collinear? Or could the coefficients/weights be used to determine relative importance of the predictors?

Oliver Angelil
  • 1,129
  • 1
  • 11
  • 24
  • 1
    the whole point of ridge regression is to fix this, even in the case of perfect collinearity. See here for example: https://stats.stackexchange.com/questions/118712/why-does-ridge-estimate-become-better-than-ols-by-adding-a-constant-to-the-diago/119708#119708 – jld Oct 20 '17 at 05:17
  • hmm, but the primary advantage of ridge over OLS is prediction accuracy? What if I'm not interested in making predictions but rather inferring what X variables are most important at explaining y. – Oliver Angelil Oct 21 '17 at 22:03
  • 1
    I think historically ridge regression was developed for solving ill conditioned systems (see https://en.m.wikipedia.org/wiki/Tikhonov_regularization) and it was only later that it was shown that there exists a $\lambda$ such that the MSE is better. So stable coefficients are really what it’s all about, although in the context of variance-bias it’s not surprising that stable coefficients lead to better predictions – jld Oct 21 '17 at 22:42
  • Cool. So I'll go ahead and use the coefficients from ridge regression to infer relative importance of the predictors. Thanks! – Oliver Angelil Oct 22 '17 at 10:41

0 Answers0