2

Using the paramater names from What is lambda in an elastic net model (penalized regression)? I wonder about best practices or suggestions for how to determine $\alpha$, i.e. the paramater determining the weighting of the Lasso and the Ridge.

My guess is that one does a 1 dimensional grid search (is there another term for that btw?) starting with 0.5 and then going of towards 0 and 1. My feeling here is that we want to not go off with the same step size all the time. I mean if one of these values is much bigger than the other perhaps we want a value really close to 0 or 1 to compensate? But how close to 0 and 1 do we test? What is the standard approach here? Is there such a thing?

Community
  • 1
jonalv
  • 153
  • 8
  • http://stats.stackexchange.com/questions/84012/choosing-optimal-alpha-in-elastic-net-logistic-regression seems very related – jonalv Jun 25 '14 at 10:46
  • You're asking about a grid search with non-uniform steps. I've never seen it done (for alpha) but if you want to investigate it, then go ahead. I'm skeptical it will give improvement. Please post some numbers if you find otherwise. – smci Jul 06 '15 at 21:51
  • I remember a nice trick from Andrew Ng's lectures: 1, 0.3, 0.1, 0.03, 0.01, 0.003, 0.001. It's approximately logarithmic but looks neat. If you don't care about your grid looking neat, just choose any logarithmic grid, i.e. start with 1 and divide by two: 1, 0.5, 0.25, 0.125, 0.0625..., or by five, or by ten. – amoeba Jul 06 '15 at 22:46

1 Answers1

3

Grid search using cross-validation is the standard approach. Like with any hyper-parameters/penalty optimization, getting a "good" grid involves much trial and error. But typically, model performance will be reasonably smooth in the parameters (i.e., you don't get wild fluctuations in cross-validated error by changing the penalties a little bit), so you can choose as big a grid as you have computing power / patience / coffee for (say 10-50 values), and that is typically enough in what I've seen.

If that's not enough, you can plot it and then focus on regions that need more exploring. Also, in a lot of real data too much tuning has not much effect on performance in external validation on other datasets, so you might not need to do that much to get the optimal performance in external validation.

The one thing to be careful with is that with alpha close to zero you will get closer to ridge regression which can be very slow if you have a lot of predictors (at least if using glmnet or similar methods), as all of them will be in the model making some of the computational shortcuts moot (active-set convergence etc).

purple51
  • 1,497
  • 10
  • 17
  • but when you say 10.50 values do you divide the span in equally sized chunks or do you take smaller absolute steps the closer to 0 and 1 you get in order to be able to test values such as 0.01, 0.001 or 0.99, 0.999? – jonalv Jun 17 '14 at 14:25
  • Yes, divide roughly equally to get an idea of the performance as a function of alpha, then near the maximum you can use a finer grid. – purple51 Jun 17 '14 at 23:37