1

For Ridge Regression, I've seen the statement that the solutions aren't equivariant to scaling of the inputs so we typically preprocess the response and regressors by standardizing.

What does "equivariant to scaling of the inputs" mean? Is OLS equivariant to scaling of inputs?

user5965026
  • 503
  • 2
  • 11
  • 1
    See the characterization of ridge regression at https://stats.stackexchange.com/a/164546/919 and consider what happens to $X_{*}$ after standardizing $X$ (the original model matrix). – whuber Jul 16 '20 at 22:02
  • @whuber I will take a look, but you mention "standardize," which is well known as subtracting the mean and dividing by the SD. Is "scaling" equivalent to "standardizing?" – user5965026 Jul 16 '20 at 22:24
  • Scaling typically means standardizing without the preliminary recentering. It doesn't matter, though: the point is that when you alter the columns of $X$ you don't alter the artificial data that Ridge Regression tacks on to them. Thus, the Ridge Regression solution depends on exactly what numbers are used to express your data, whereas the OLS solution does not. – whuber Jul 17 '20 at 14:41
  • @whuber Ah so my understanding is for OLS, if you scale the input X, you can simply just scale the estimators by the inverse of the scaling of X. However, you can't do this with Ridge because if you look at the solution for ridge, there's a $+ \lambda I$ term in the matrix you're inverting. If the scaling was also applied to the regularization term, then ridge regression would be equivariant. – user5965026 Jul 17 '20 at 16:53
  • Yes, that's right. Another way to put it is that the strength of the regularization, as reflected in the magnitude of $\lambda$ *relative* to the magnitude of each column, depends on how you scale each column. – whuber Jul 17 '20 at 17:48

0 Answers0