1

I wish to apply 10 fold cross validation to my model (with a sample size of 90). I have one question troubling me.

Once I am ready with a full model, I would break the full data into 9:1 ratio- 9parts for training and 1 part of testing, and in the same way I would be testing all the parts in 10 rounds.

The question that is troubling me is that how would I take the variables in each of the rounds. The variables would not be same.

If I am not correct please give me the steps to choose the variables in this stage?

Arvinder
  • 11
  • 3

1 Answers1

1

CV is a method for estimating the performance of a method for fitting a model, not the model itself. So use CV to get a performance estimate, and then retrain the model on the full dataset to get the model used in operation.

If different variables are selected in each fold, that is an indication that the feature selection is unstable/unreliable and you would probably be better off using a regularised model (e.g. ridge regression) instead of feature selection.

See also my answers to some related questions here (the last link is probably the most relevant).

Community
  • 1
Dikran Marsupial
  • 46,962
  • 5
  • 121
  • 178
  • What I have done in order to arrive at the multiple regression equation....I have selected some predictor variables (18 variables both dichotomous and continuous) and created a sample size of 150. Then applied DFit test and removed the outliers in Minitab one by one, even I was able to delete some variables which were insignificant and finally arrived at the regression equation with 6 predictor variables. Is this a correct way? Now I want to do 10 fold cross validation. If I am missing something please advise? – Arvinder Dec 01 '14 at 14:33
  • 1
    The details of the analysis depend on the aims. The important thing is that in CV you need to repeat all of the steps you performed to make the final model independently in each fold of the cross-validation procedure. It is a technique for evaluating a method of generating a model, so you need to repeat all of the same steps used to create the model. Whatever you do, don't determine the features and then cross-validate using that set of features, or you will have an optimistically biased performance estimate. – Dikran Marsupial Dec 01 '14 at 14:51
  • Thank you for the information and I am really thankful to you for addressing my query. Regarding " Whatever you do, don't determine the features and then cross-validate using that set of features, or you will have an optimistically biased performance estimate", this means I need to first of all divide the full data set in the 90:10 ratio then determine the features for all the folds ....is it possible that training and test data may have different features? My aim is prediction. – Arvinder Dec 01 '14 at 16:23
  • yes, that is correct, the paper by Ambroise and McLachlan that I mentioned in one of my answers explains this issue very clearly. – Dikran Marsupial Dec 01 '14 at 16:51
  • if we have worked around 10 folds one by one, what should we do next? Could you please suggest me a step by step link? – Arvinder Dec 01 '14 at 17:30
  • I would like to know that in the end the final data set that we would be using for regression equation is the same equation that we found before applying 10 fold cross validation? Also, MSE of the 10 test set we have obtained, what to do with that. Please clarify? I am a non statistician but willing to work in this area. – Arvinder Dec 02 '14 at 10:16