21

I am learning about ridge regression and know that ridge regression tends to work better in the presence of multicollinearity. I am wondering why this is true? Either an intuitive answer or a mathematical one would be satisfying (both types of answers would be even more satisfying).

Also, I know that that $\hat{\beta}$ can always be obtained, but how well does ridge regression work in the presence of exact collinearity (one independent variable is a linear function of another)?

TrynnaDoStat
  • 7,414
  • 3
  • 23
  • 39
  • 5
    Regarding your second question: If you have exact colinearity you can just remove one of the variables. You don't need ridge regression. – Peter Flom Jun 25 '14 at 20:29

1 Answers1

21

Consider the simple case of 2 predictor variables ($x_1$, $x_2$). If there is no or little colinearity and good spread in both predictors, then we are fitting a plane to the data ($y$ is the 3rd dimension) and there is often a very clear "best" plane. But with colinearity the relationship is really a line through 3 dimensional space with data scattered around it. But the regression routine tries to fit a plane to a line, so there are an infinite number of planes that intersect perfectly with that line, which plane is chosen depends on the influential points in the data, change one of those points just a little and the "best" fitting plane changes quite a bit. What ridge regression does is to pull the chosen plane towards simpler/saner models (bias values towards 0). Think of a rubber band from the origin (0,0,0) to the plane that pulls the plane towards 0 while the data will pull it away for a nice compromise.

Greg Snow
  • 46,563
  • 2
  • 90
  • 159
  • @Trynna, there are [pictures](http://stats.stackexchange.com/a/70910/3277) illustrating what Greg said about collinearity problem. – ttnphns Jun 26 '14 at 06:14
  • 2
    This is a very good geometrical explanation about why multicollinearity is an issue in OLS regression! But I still don't quite understand why pulling the plane to the origin fixes the problem. – TrynnaDoStat Jun 26 '14 at 13:52
  • 3
    @TrynnaDoStat, The main concern is the variability of the estimates, with the multicolinearity, a small change in a single data point can wildly swing the coefficient estimates (without the bias). By biasing towards 0 there is not much change in the estimates of the coefficients (because that rubber band is pulling them towards 0) with a minor change in a single data point, reducing the variability. – Greg Snow Jun 26 '14 at 13:58
  • 1
    Thanks @ttnphns for the link to the pictures: without it it was pretty a stretch to get the answer. Now Greg's answer is clear and what I needed to understand this line in ESLII(2nd ed.): "wildly large positive coefficient on one variable can be canceled by a similarly large negative coefficient on its correlated cousin. By imposing a size constraint on the coefficients this problem is alleviated." – Tommaso Guerrini May 06 '17 at 14:08