9

I am aware that CV was born as a way to validate models when there is a lack of training data, but my understanding is that it is generally better to cross validate rather than just use one validation set as this gives a more unbiased model selection step and reduces randomness in model results due to the selection process of the validation data.

Aside from increased computational expense, are there any other drawbacks to cross validation compared to normal validation? Is it safe to say that, if computational complexity weren't an issue, one would always be better off cross validating rather than just using normal validation?

Frans Rodenburg
  • 10,376
  • 2
  • 25
  • 58
Odisseo
  • 614
  • 5
  • 11

2 Answers2

10

Cross-validation as an alternative in case of a lack of training data is quite an understatement. Unless your sample size is very large, validation performance can vary widely among different random splits.

Cross-validation suffers less from this, since it considers the results from multitude folds. Even better would be to average over multiple runs of cross-validation, each with a different random split into $k$ folds.

What you consider ordinary validation is really just cross-validation with a single fold. Examples of where you might intentionally want to use $k=1$ over multiple folds include:

  • Your cannot afford $k>1$ computationally;
  • You have, say, millions of records and can confidently split your data randomly;
  • You are performing external validation and want to demonstrate that your model still performs well on data from a source never before seen by the model.

In case of the latter, your model would likely generalize better if you include data from multiple sources for training (e.g. data from different institutes, studies, or data bases). However, if you use all sources for training, you still don't have a real estimate of the performance on a new source.

Frans Rodenburg
  • 10,376
  • 2
  • 25
  • 58
2

Another drawback of CV (in addition to those in Frans Rodenburg's excellent answer) is when there is a dependency between samples, such as in a time series. In that case you can split into train/valid such that no training data has a dependency on any validation data. (E.g. in the case of time series, your validation data has the later timestamps.)

Darren Cook
  • 1,772
  • 1
  • 12
  • 26