3

In ridge regression using unnormalized features, if you double the value of a given feature A (i.e., a specific column of the feature matrix), what happens to the estimated coefficients for every other feature?

My understanding is that the weight of A is halved, and because of L2 regularization, the concentric circles become ellipses, and all the coefficients associated with other features should all be halved. In the limit of feature scaled to infinity, all the other features become irrelevant (which is why standardization is important).

However, my answer is wrong, and please kindly rectify my reasoning and point out the solution process for the rest of us. Thank you!

Frank
  • 63
  • 1
  • 7
  • 2
    What is your understanding of this situation for ordinary least squares regression? (Since Ridge Regression [can be formulated in terms of OLS regression,](http://stats.stackexchange.com/a/164546/919) this is worth thinking about.) – whuber Mar 30 '16 at 20:41
  • Hi whuber, thanks for the comment. In case of ordinary least squares regression, coefficients associated for every other features should remain unchanged, while the weight of feature A is halved. However, in case of ridge regression, my intuition is limited in this regard: (1) as what I have pointed out in the above; (2) if features have some collinearity or part of the columns are strongly correlated, then not all weights are affected in the same way, i.e. half reduction. Am I right? Thanks for the link, I will take a look. – Frank Mar 30 '16 at 21:38
  • 1
    Because your intuition for ridge regression is so far away from what actually happens in OLS, something must be the matter. After all, OLS is the limit of ridge regression as the parameter goes to zero and (except in the case of perfect collinearity) the ridge traces are *continuous*. Thus, what you know about OLS ought to hold to a good approximation for small values of the ridge parameter (at least). – whuber Mar 30 '16 at 21:40
  • Thanks. So, regarding my question, how do we understand it? – Frank Mar 31 '16 at 00:16
  • I would be curious to have a rigorous answer to this question, as I feel it depends on whether the features are perfectly correlated, uncorrelated, partially correlated, normalized ? – Xavier Bourret Sicotte Jun 08 '18 at 14:11

1 Answers1

1

In case of unnormalized features in ridge regression, the coefficients of other columns may also change depending upon the correlation between them. We can't tell about the coefficients of other columns. While in case of normalized features in ridge regression, the column whose values are doubled gets the coefficients halved while others remains unchanged.

Gaurav Kumar
  • 11
  • 1