0

Supposed I have a small number of features, say 4 or 5, and I have hundreds of data points. That is, I am in an over-determined situation.

Is there any benefit to using regularization in this setting or is it standard practise to just use least squares?

If there is a benefit to it in the over-determined case, what is the intuition of what is happening when we increase/decrease the regularization parameter?

ManUtdBloke
  • 743
  • 4
  • 13

1 Answers1

3

If you are interested in predictive power then you could definitely look at regularization coupled with adding more complexity such as interaction terms or whatever gives you best cross validation accuracy.

If you are interested in inference then I would probably stick to OLS since we are biasing our coefficients and we lose the ability to calculate consistent variance estimates of those coefficients. That is, unless we do it via a bayesian methodology.

Regularization as a whole is NOT just for instances where our number of variables are greater than observations. Instead, it can be great for balancing the bias-variance tradeoff and keep our model from choosing coefficients which 'overfit' our current data and instead choose coefficients which generalize better to new data.

The intuition can be seen nicely when looking at level curves where the two dimensions are all possible values of 2 coefficients for 2 variables, the blue circle is the possible coefficients to choose from a ridge constraint. https://i.stack.imgur.com/s2Iey.png

The ridge constraint keeps us from actually finding the 'optimal' values but if the green circles were to move then the coefficients chosen via ridge would not move nearly as much as the ols coefficients would.

For how the regularization parameter effects what happens it simple is just effecting how 'big' that circle is.

Tylerr
  • 1,225
  • 5
  • 16
  • 1
    (i) You say we lose the ability to calculate consistent variance estimates if we use regularized least squares. Could you be more specific about what what we can with OLS that we can no longer do with regularized least squares? – ManUtdBloke Aug 22 '20 at 18:27
  • 1
    (ii) Would you have a link to a paper/book where I can see an example of regularization being used to balance the bias-variance tradeoff in and overdetermined setting? – ManUtdBloke Aug 22 '20 at 18:27
  • 2
    Here is a similar question about Lasso (L1 regularization) and standard errors and there are a lot of good resources linked in it: https://stats.stackexchange.com/questions/91462/standard-errors-for-lasso-prediction-using-r . – Tylerr Aug 23 '20 at 01:32
  • 1
    I don't have anything handy for regularization and bias-variance. But that is the point of regularization - add bias in order to reduce variance. Definitely run some simulations with train-test data splits and you can see it in action. We can have some issues with linear models where our model isn't really 'low bias' so adding regularization just does nothing. In those scenarios we usually reduce bias by adding interaction terms or polynomial expansions of our variables then regularization can increase that bias to decrease the variance. – Tylerr Aug 23 '20 at 01:48
  • 1
    http://faculty.marshall.usc.edu/gareth-james/ISL/ – Christoph Hanck Sep 02 '20 at 12:51