0

I happened to an interview question:

In Ridge regression, what does it imply if the out-of-sample performance never change however we tune the hyperparameter (the coefficient of L2 regularization)?

I only know that there exists an optimal coefficient which makes the MSE of ridge smaller than its OLS estimation. I am not sure if it is related to this question and I have the no idea for the answer.

Sycorax
  • 76,417
  • 20
  • 189
  • 313
user6703592
  • 745
  • 3
  • 8
  • What is L2 regularization intended to accomplish? Think, then, of what it can accomplish when that objective is unnecessary: namely, when all the explanatory variables already are orthogonal. – whuber Jul 30 '21 at 20:00
  • @whuber does that mean when the multicollinearity is not heavy, change of $\lambda$ cannot improve the performance (reduce MSE) significantly? Here mentioned the `out-of-sample`, why do we specially use this? – user6703592 Jul 30 '21 at 20:30
  • That is the heart of the question. It makes one wonder exactly what "out-of-sample performance" really means. Exactly what observations constitute "out of sample"? Would it be referring to cross-validation? Testing on a hold-out set? Testing on new data altogether? Once you have decided that, it would be possible to analyze the situation mathematically. I find the approach described at https://stats.stackexchange.com/a/164546/919 to be particularly helpful at visualizing what might be going on. – whuber Jul 30 '21 at 20:48
  • @whuber many thanks. And could you also refer me some answers of "why ridge can reduce the multicollinearity"? Since up to now, I can only explain: `multicollinearity increases estimation variance -> ridge reduces variance -> therefore ridge reduces multicollinearity.` But actually this logic does not convince me strongly, It is better to have some detailed deduction. – user6703592 Jul 31 '21 at 06:06
  • @whuber very nice explanation of extended $n+p$ matrix. But i still don't understand "out-of-sample performance" here, could you open an answer here to discuss more details? Since recently I happened too many questions of ridge and multicollinearity. – user6703592 Jul 31 '21 at 07:05

0 Answers0