1

Why do we use Ridge regression instead of Least squares in Multicollinearity?

Which one is correct:

a. lower bias and higher variance
b. lower bias with the same variance
c. higher bias with a lower variance
d. the same bias with lower variance

amoeba
  • 93,463
  • 28
  • 275
  • 317
Mike
  • 21
  • 1
  • 4
  • Which is correct: I should eat an apple, or I should eat an orange? They are both correct, for different situations. Sometimes you don't mind a bit of bias, sometimes you do. – Jeremy Miles Jan 29 '17 at 19:36
  • 5
    Is this an exercise/exam question? If so, see http://stats.stackexchange.com/tags/self-study/info – Juho Kokkala Jan 29 '17 at 19:42
  • 1
    @JeremyMiles, in this situation the answers are not equally correct, because the situation is well defined: multicollinearity. – Richard Hardy Jan 29 '17 at 20:00
  • Oh, thanks. You're right, I should have read the question more closely. – Jeremy Miles Jan 29 '17 at 20:06
  • You do not need to put [solved] in the title. There is an accepted answer, it means that this is solved. – amoeba Jan 30 '17 at 11:25

2 Answers2

3

Your OLS estimator is $$ \hat{\beta}_{ols} = (X'X)^{-1}X'y, $$ while your ridge regression estimator is $$ \hat{\beta}_{ridge} = (X'X + \lambda I)^{-1}X'y. $$ Take the expectation and variance of each one, and then compare your results.

Taylor
  • 18,278
  • 2
  • 31
  • 66
2

Why would we use ridge regression? It's useful because it prevents overfit to your training data. Now, ask yourself how does preventing overfit influence bias and variance.

I have a useful link for you: When to use regularization methods for regression?

enter image description here

Community
  • 1
SmallChess
  • 6,764
  • 4
  • 27
  • 48
  • ok got it! It's higher bias with a lower variance ! Right @student T? – Mike Jan 30 '17 at 11:06
  • @Mike Yes. You learn. – SmallChess Jan 30 '17 at 11:10
  • @Mike It's time for you to appreciate our efforts. Please accept an answer, myself or Taylor. This is an opportunity for you to appreciate our efforts. You should see a green tick near our answers. Please pick one. – SmallChess Jan 30 '17 at 11:17