1

I know there are many questions about cross validation but I couldn't find that generally describes all aspects (I have in my mind). In How to evaluate the final model after k-fold cross-validation it is described that cross validation is used to select the best model. I wonder about the following:

  1. Why are different models not just trained on the same data set and tested on the same separated dataset? Why the effort for using a different data set for each model?
  2. The same applies when searching the best parameters for a single model (like in grid search).

Can you please clear my questions?

CV

Ben
  • 2,032
  • 3
  • 14
  • 23
  • 1
    In my opinion, train/test approach gives you a point estimation which may be terribly wrong due to chance, but cross-validation is giving a distribution. Which is always better. – German Demidov Oct 09 '19 at 08:08
  • Thank you. But how exactly is it used? – Ben Oct 09 '19 at 08:10
  • Well, you select the fold-size according to your sample size, run cross-validation, look at the distribution of test performance and make conclusions =) like, how stable is the performance, what is the expected performance, etc. I think there are plenty of manuals on this. – German Demidov Oct 09 '19 at 08:27
  • 1
    *Why are different models not just trained on the same data set and tested on the same separated dataset? Why the effort for using a different data set for each model?* - I think you might be wrong here. Practitioners do train different models on the same data set and compare their performances . That's the approach when you want to select the best model for the given data. – naive Oct 09 '19 at 10:48
  • 1
    @GermanDemidov CV also returns point estimates, which may be even "wronger", since they are estimated on less data. Would you rather have 5 estimates of the mean from 5 smaller samples or 1 estimate from a larger sample? OP: what naive said, different models are built on the same data and tested on the same data. – user2974951 Oct 09 '19 at 11:03
  • @user2974951 I can perform leave-one-out cross validation if my sample size is small. CV returns *number of folds* measures of performance - isn't it different from point estimate? Like, when we do bootstrap, is it point estimate what we get in the end? I think no, and I do not see a fundamental difference between CV and bootstrap in this sense. Like, this plot http://www.milanor.net/blog/wp-content/uploads/2016/05/articolo_milanor_3.png clearly has confidence intervals. – German Demidov Oct 09 '19 at 12:00

1 Answers1

1

Writing an answer as answering in the comments is not really desirable.

Why are different models not just trained on the same data set and tested on the same separated dataset? Why the effort for using a different data set for each model?

  • I think you might be wrong here. Practitioners do train different models on the same data set and compare their performances . That's the approach when you want to select the best model for the given data.

How to perform cross-validation?

Have a look here.

naive
  • 899
  • 1
  • 9
  • 14