4

I would like to do model selection using backward stepwise procedure and cross validation. https://www.otexts.org/fpp/5/3

I have used stepAIC in MASS package for predictors selection and I would like to see whether cross validation is a better criteria. In other words, I am trying to use cross-validation as a criterion in backward selection procedure instead of AIC.

The procedure should start with a full model, drop one predictor at a time, keep the variable if it improves CV and continue until no improvements can be made.

I think that the rfe function in the caret package may be able to do the job. I would also like to do bootstrapping if possible. If so, how should I specify it? Or are there any functions that can do that? I have tried writing my own function, but it seems to be beyond my ability. Many thanks.

AdrienS
  • 63
  • 1
  • 6
  • 1
    `validate` in the `rms` package has the option of allowing backwards selection based on AIC. Do read some of the many posts here on why stepwise methods aren't so good & consider whether (a) you need to do post-hoc model selection, & (b) you ought to apply shrinkage e.g. lasso. – Scortchi - Reinstate Monica Feb 21 '14 at 12:03
  • 4
    As @Scortchi alluded, the method you proposed is not based on any good statistical principles. If you really need to do model selection (which is not obvious why this is helpful; weak variables don't hurt models), then you should combine model selection with shrinkage using a method such as elastic net. – Frank Harrell Feb 21 '14 at 12:34
  • 1
    Sorry, I misread the question (despite its being pretty clear) - you want to use a performance measure calibrated by cross-validation as a criterion in the backward selection procedure *instead* of AIC. Note that you'll still want to validate this procedure: see [here](http://stats.stackexchange.com/questions/65128/nested-cross-validation-for-model-selection). And that it's still stepwise, with all its faults. – Scortchi - Reinstate Monica Feb 21 '14 at 12:35
  • 2
    In addition to what Frank Harrell and Scortchi say, you should definitively read Dikran Marsupial's paper about overfitting during such massive model comparison situations: Cawley, G. C. & Talbot, N. L. C.: On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation, Journal of Machine Learning Research, 11, 2079-2107 (2010). – cbeleites unhappy with SX Feb 21 '14 at 13:19
  • 1
    Thanks for the comments. As @Scortch clarified, I am trying to use cross-validation as a criterion in backward selection procedure instead of AIC. I have used backwards selection based on AIC and I would to see whether cross-validation is a better criteria. – AdrienS Feb 24 '14 at 03:00
  • See [this page](http://caret.r-forge.r-project.org/Feature_Selection_Wrapper.html) for details on feature selection wrappers used by `caret` via `train`. Using `method = 'lmStepAIC'` or `method = 'glmStepAIC'` will resample that entire process (if you are interested in that sort of thing) – topepo Feb 24 '14 at 15:54

0 Answers0