5

Let's say I have 20 different datasets, some of which are as small as 200 data items and some have around 2000 samples. The goal of analysis is to do some sort of performance comparison between the models working on each dataset.

Now say for the smaller datasets I've gone with the LOOCV for performance estimation but it's computationally impossible to do LOOCV for the larger ones. Then would the choice of 100 be reasonable over the generic value of k in k-fold CV, just to make the analysis more consistent across the datasets?

In general, I've seen debates on 10-fold vs LOO CV on considerations such as variance-bias trade-off, but I'm wondering if there's any case that a k value like 50 or 100 would be preferable to those generic values we commonly see in discussions.

Aliweb
  • 205
  • 2
  • 10
  • 2
    I don't quite understand your concern. k=10 is reasonable. k=N (where usually N>100) is reasonable. So why would k=100 be unreasonable? – amoeba Jan 25 '17 at 18:01
  • 2
    In a certain sense, this reduces to a question of bias-variance tradeoff in the estimates of model performance. http://stats.stackexchange.com/questions/61783/variance-and-bias-in-cross-validation-why-does-leave-one-out-cv-have-higher-var – Sycorax Jan 25 '17 at 18:03
  • @amoeba because k=10 is usually introduced as a middle ground and an optimal point according to variance-bias tradeoff. – Aliweb Jan 25 '17 at 18:05
  • I deleted my previous answer as I was not really aware of the variance possibly introduced via LOO. Thanks @amoeba for pointing it out. – Martin Krämer Jan 25 '17 at 18:08

0 Answers0