3

I would like to perform $K$-fold cross-validation on a dataset to test generalization of a model.

For about the same amount of computational effort, I could perform one full cross validation with say $K = 20$, or two cross-validations with $K = 10$ but playing around with completely different partitioning schemes for the data.

Is there any reason to prefer the former to the latter?

In other words, is increasing $K$ always a good thing?

As an argument against it, I can imagine that leaving out a good chunk of the data (possibly by partitioning the data in a smart way, and trying out different ways of partitioning) might help test extrapolation to unseen regions.

lacerbi
  • 4,816
  • 16
  • 44

1 Answers1

2

This depends on the sample size. Larger K reduces bias with respect to overestimating the true generalization capability (as training dataset will be more and more similar to original dataset) but results higher variance and higher running time, as you have noticed. This is because we get closer and closer to Leave-One-Out CV. The choice. The choice of k is arbitrary but k is often chosen as 10 to balance bias and variance.

Your Questions

  1. Is there any reason to prefer the former to the latter?

    I would prefer the later because it will expected to lower the variance of cross-validation, and the method is often known as repeated cross-validation. For details refer to this question Choice of K in K-fold cross-validation

  2. In other words, is increasing K always a good thing?

    No because of the answer in this question Optimal number of folds in $K$-fold cross-validation: is leave-one-out CV always the best choice?

Community
  • 1
discipulus
  • 726
  • 4
  • 14