0

I am coming from the field of psychology and in most publications Model Selection (OLS, Regression) is done via Forward/Backward Selection using the F-Static/p-value of the regression coefficients to decide which coefficient to include/exclude in the next step.

Now i read in the statistical learning literature (for example in ESL) that this strategie is outdatet because "it doesn't account for multiple testing issues correctly".

Unluckily there is no further explanation on those problems.

Does anybody can explain me, what exactly is the problem of this approach and why using other criteria (Cp, AIC) is better?

Thanks a lot in advance!

platypus
  • 66
  • 6
  • Using Cp or the AIC is not necessarily much better. – gung - Reinstate Monica May 21 '20 at 15:59
  • 1
    I think you will find the information you need in the linked thread. Please read it. If it isn't what you want / you still have a question afterwards, come back here & edit your question to state what you learned & what you still need to know. Then we can provide the information you need without just duplicating material elsewhere that already didn't help you. – gung - Reinstate Monica May 21 '20 at 16:00
  • @gung-ReinstateMonica Okey i read trough the post. I hope, that i unterstand your answer their correct. You want to tell us, that the problem with data driven model building is, that they are highly unstable - using a different sample leads to different models (at least when the n/p ratio isn't extraordinary high). Using for example Cross Validation at least reduce risk of overfitting, but Prediction Error itself is an estimator and depends on the data set/the split used for CV. What i did not get is what do i do with this situation now? When i have to do model building - is their any way out? – platypus May 21 '20 at 21:13
  • 1
    it isn't just overfitting / poor out of sample predictive performance. These methods are highly unlikely to select the right variables. The hypothesis tests are invalid. Etc. To a first approximation, there is nothing good about stepwise selection. – gung - Reinstate Monica May 22 '20 at 20:01
  • 1
    Regarding the "way out", there isn't really one. This is a very difficult problem. For the most part, you want to think hard, before you conduct a study, about what variables you want to include & why. Ie, variable selection is done *before the data exist*, & is done *based on your knowledge of the topic*. If you don't have much relevant knowledge, go get some. Talk to others in the field; read the literature. Etc. You can figure out how much data you can get & how many variables you can afford, & then include a bunch--there's no problem if you have non significant variables in your model. – gung - Reinstate Monica May 22 '20 at 20:07
  • 1
    After all that, if you *really* need to do variable selection (ask yourself why you need to do this), you can give up on the idea of hypothesis tests, and use cross-validation and penalization methods. – gung - Reinstate Monica May 22 '20 at 20:10
  • 1
    @gung-ReinstateMonica thanks for the advice! i was thinking about that now for a couple of days and now i have the feeling that i know more what i am doing. In most cases i am going to build my model based on psychological background knowledge and use ridge regression as some kind of finetuning of the prediction including CV for the hyperparameters. In some cases this is not possible - for example when working with genetic data or a neuro/imaging data - in this cases i will try to refrain from model seletion as long as my n isn't at least as big as my p. – platypus May 24 '20 at 22:33
  • You're welcome, @platypus. Good luck with your research. – gung - Reinstate Monica May 25 '20 at 11:49

0 Answers0