0

L1 regularization reduces the magnitude of weights but does not make them zero. Is there a way we can make them zero. Will it be better than normal L1 regularization

shaifali Gupta
  • 101
  • 5
  • Increase the penalty, regularization parameter enough and all the coefficients will be zero. – Matthew Gunn Aug 16 '17 at 17:55
  • @MatthewGunn What is the maximum value of parameter. Is it between 0 and 1? – shaifali Gupta Aug 16 '17 at 18:06
  • 1
    I assume you're referring to [LASSO](https://en.wikipedia.org/wiki/Lasso_(statistics)) regression? If so, there's no upper bound on the regularization parameter. – Matthew Gunn Aug 16 '17 at 18:33
  • @MatthewGunn Then how to choose the value of regularization parameter. because there is no limit on the number of cases that can be tried – shaifali Gupta Aug 18 '17 at 14:37
  • Perhaps look through some tutorials such as [here](http://scikit-learn.org/stable/auto_examples/exercises/plot_cv_diabetes.html#sphx-glr-auto-examples-exercises-plot-cv-diabetes-py) or the Machine Learning class by Andrew NG on CourseRA (or similar). Generally, the error is a convex function of the regularization parameter $\alpha$ (or equivalently, the cross validation score is a concave function of the parameter). People generally do some kind of grid search or randomized search to find a parameter value that works reasonably well using performance as estimated on the validation set. – Matthew Gunn Aug 18 '17 at 15:09
  • @MatthewGunn Do you perhaps have a reference for the claim that the test error is a convex function of the regularization parameter? I've tried searching for a result like this in the literature with no luck. – user795305 Sep 05 '17 at 17:47
  • 2
    @Ben Looking back, I shouldn't have written that. In practice, it often looks convex (or quasiconvex) but [there's no requirement](https://stats.stackexchange.com/questions/298509/is-the-error-rate-a-convex-function-of-the-regularization-parameter-lambda). – Matthew Gunn Sep 05 '17 at 18:30
  • Laso, which is L1, does do variable selection? https://stats.stackexchange.com/questions/74542/why-does-the-lasso-provide-variable-selection – kjetil b halvorsen Oct 26 '21 at 18:26

0 Answers0