What value of alpha should I choose in glmnet? Should I use one which minimizes the cross-validation error, one which is one standard deviation above or below the one which gives the best error (like cp for decision trees)? What is the common practice?
Asked
Active
Viewed 225 times
1 Answers
0
The answer partly depends on the point of the exercise - I.e. what you want to get out of it - and partly on what your data looks like.
If the goal of the exercise is predicting to new data, rather than just identifying important links then using a metric which gives you an accurate evaluation of model performance is a logical way forward. In this case it is sensible to calculate prediction error through either cross-validation or boot-strapping with the appropriate adjustment (i.e. + 0.632 - see this for more information).
The second part of the problem is which metric is appropriate; there are many. Root mean square error (RMSE), median absolute error (MdAE a.k.a. median absolute deviation (MAD)), some variation of an asymmetric loss function, etc. RMSE is really commonly used but since it take the mean of the squared errors it is sensitive to outliers, something which MdAE is robust too - this is where your data comes into the mix. It also somewhat depends on the question of interest. If some errors are worse that others (i.e. under-prediction is far worse than over-prediction) then you can weight your errors through a asymmetric loss function, although there isn't a one size fits all solution in this case.
If you post some plots of your residuals and a little more information on your problem of interest then I might be able to say more, although I hope this helps!
EDIT: Common practice, I believe, is to use CV or Bootstrapping + .632 to optimize alpha by minimizing prediction error.
André.B
- 1,290
- 6
- 20