4

I am using the least angle regression (LARS) to extract the most important predictors ($x_1, x_2,...,x_p$) for my response variable ($y$).

I have seven predictors ($x_1,x_2,...,x_7$) for each response variable. I did 10-fold cross validation by using R package lars.

I am making a cut point at 0.62 (approx.). Am I doing right? Is there any criteria to select important predictors?

Any help in this regard would be highly appreciated! Thanks!

Biostat
  • 1,791
  • 2
  • 19
  • 21
  • Any reason for removing the inset figures? The current answer makes references to the figures before the edit, so it may confuse future readers. – syockit Sep 24 '21 at 04:31

1 Answers1

6

First, yes, you are following a standard way to select the model by choosing the model that corresponds to $0.62$ on the $x$-axis. Second, it depends what you mean by "important predictors", but it seems that you just used a criteria to select predictors. The right plots suggest 2-3 predictors are selected for the models that correspond to the $\sim 0.62$ value.

The model chosen is based on the "one-standard error rule-of-thumb". This rule can be tracked to the 1984 CART book by Breiman et al., and it says that when you do cross-validation for model selection, you should not choose the model that obtains the lowest value of the estimated expected generalization error, but you should take a model that is as simple as possible and with an estimated expected generalization error within 1 standard deviation from the lowest estimate obtained. It's a conservative choice that has some empirical support and perhaps some qualitative theoretical support, but why it should be precisely one standard error is ad hoc.

The resulting model is primarily chosen to optimize performance of the model as a predictive model. The lasso penalization provides a combination of shrinkage of parameter estimates and parameter selection, and in this tradeoff for optimization of predictive performance, lasso is known to generally choose too many predictors. Thus don't expect that all the predictors chosen by a lasso procedure, as implemented in lars, are important. Moreover, even if they are important, they are important as predictors in the given regression setup, and can not automatically be given any causal interpretation.

I would, by the way, recommend glmnet over lars. It's a faster implementation and a more flexible class of models and penalization functions.

NRH
  • 16,580
  • 56
  • 68
  • Thanks for useful reply! Could you please explain more about this rule "you should take a model that is as simple as possible and with an estimated expected generalization error within 1 standard deviation from the lowest estimate obtained." Can I generalize the "one-standard error rule of thumb" in mathematical form i.e., does it make sense $min(cv_xbar-cv_stderr)$? or you're saying something different? Please note that "important predictors" means just same selecting predictors. – Biostat Oct 30 '11 at 10:24
  • @biostat, I don't understand your math expression. I'm not sure that it will become more clear in a mathematical formulation. It's just a rule-of-thumb saying that you will, at least, not loose anything in terms of predictive performance by choosing a simpler model within one standard deviation from the model with the smallest estimate of the expected generalization error. So why not choose the simplest such model? With lasso, models with a larger penalty (to the left on the plot) will generally be simpler (have less non-zero estimated coefficients). – NRH Oct 30 '11 at 19:19
  • Sorry for bad notations! it was just by mistake. If you know any literature about these criteria for CV, please let me know. Thanks! – Biostat Oct 31 '11 at 01:15
  • 1
    @biostat, [ESL](http://www-stat.stanford.edu/~tibs/ElemStatLearn/), but I believe that the original (written) reference to this rule-of-thumb is the Breiman et al. book I refer to in the answer. – NRH Oct 31 '11 at 08:04
  • Let denote $cv$ is 10 fold CV prediction error and $cv_m$ the mean of that 10 fold CV prediction error. What would be the standard error $cv_s$ of that prediction error? Is it $SD(cv_m)$ ? What do you mean by "lowest estimate obtained" in your first comment? I am little bit confused, please clarifiy! Thanks! – Biostat Nov 01 '11 at 11:46