0

Question: How is K-fold cross validation better than an implementation of 'grid-search' with repetitions when tuning hyper-parameters in a model?

Context: I am tuning hyperparameters in a model. I have implemented grid search in such a way as to try out different combinations of parameters, repeat each combination multiple times, and then average the results. For each repetition, the training and validation sets are being completely regenerated from scratch (taking the whole set and doing the splitting again). If I now am told to implement K-fold cross validation, how do I know which of these two methods will yield better results (and why)?

I understand that k-fold cross validation is meant to decrease the variance that we might see by doing a basic grid-search implementation whereby we just trial each parameter combination only once, but what about when each combination has been repeated? By repeating each combination more than once, have I implemented cross validation incorrectly? Including the repetitions seemed intuitive as the one trial seemed un-reliable.

Is a grid-search with k-fold cross validation likely to be better than a grid-search w/ repetitions but no k-fold cv?

Rocky the Owl
  • 21
  • 2
  • In the repetition case, how you split your data set into training and validation, in an arbitrary way or in a systematic one ?? – Fiodor1234 Dec 11 '20 at 17:35
  • 1
    Does this answer your question? [K-fold vs. Monte Carlo cross-validation](https://stats.stackexchange.com/questions/51416/k-fold-vs-monte-carlo-cross-validation) – Ryan Volpi Dec 11 '20 at 18:56
  • Assuming you split the data randomly each time you create train-test sets, that method is called Monte Carlo cross-validation. See the post linked above for a comparison of K-fold and Monte Carlo cross-validation. – Ryan Volpi Dec 11 '20 at 18:58
  • @Fiodor1234 - I split it by getting the list of indices for the whole dataset and selecting a random permutation consisting of the required numbers for the train set and then the rest were for the testing set – Rocky the Owl Dec 11 '20 at 19:03
  • @RyanVolpi - thank you very much, that is exactly what I have done. However, if we have results from Monte Carlo cross validation and K-fold cross validation, how could we go about choosing which one to use? Would one just choose the results that yield the lowest (average) metric? – Rocky the Owl Dec 11 '20 at 19:04
  • These are simply two methods of estimating the model's future performance on unseen data. They don't effect how the model will actually perform so there is no reason to prefer the one which gives lower estimates. However, estimating both and choosing the lower one is not a good idea as it will result in a estimate which is biased low. Just choose one method (perhaps based on the pros/cons discussed in the linked question) and stick with it. In my opinion, the choice is of little consequence, so don't stress about it. – Ryan Volpi Dec 11 '20 at 20:02

0 Answers0