4

I'm using 5-fold cv for parameter optimisation in a regression problem. I have very few samples: around 50.

Should I use leave-p-out cv instead? (with, say, p=5)

What are the (theoretical, ignoring performance) reasons to choose one over the other?

Fabio
  • 270
  • 2
  • 8

1 Answers1

2

Note:
leave-5-out CV = 10-fold CV for $n = 50$
leave-10-out CV = 5-fold CV for $n = 50$

For the choice of $k$/$p$ please see Choice of K in K-fold cross-validation

Update: exhaustively testing all possible splits

I suggest that you read up about iterated/repeated cross validation which fills the gap between testing each sample once and testing all possible splits exhaustively.

Testing each sample more than once allows to measure the stability of the predictions wrt. slight changes in the training data. But it obviously does not increase the number of independent test cases.
Thus, random error due to model instability is reduced, but the random error due to the finite (small) number of test cases is not affected.

Community
  • 1
cbeleites unhappy with SX
  • 34,156
  • 3
  • 67
  • 133
  • 3
    That's not true: leave-p-out is exaustive, k-fold is not. So for example leave-5-out for 50 samples means CV will have 2118760 iterations (all possible 5 elements are, in turn, used as validation set). 5-fold instead is only 5 iterations (the data is split into five equally-sized blocks and each block is, in turn, used as validation set). That's why the latter is usually preferred - but if it's worth it on theoretical grounds, I'm happy to take the performance hit (leave-2-out most likely, though). Hence my question. – Fabio Oct 13 '15 at 14:57
  • 1
    Fabio, can you point me to some reference where leave-$p$-out is defined as going through all possible splits? So far, I've encountered it only as "in turn, leave-$p$-cases out until each sample was left out once". – cbeleites unhappy with SX Oct 14 '15 at 13:28
  • 1
    http://www.sciencedirect.com/science/article/pii/S0167947307003842 Section: "Definition 2.1 LPO risk estimator" It's not often quoted (you usually only see leave-1-out compared to k-fold) but is technically "better". That is, k-fold is an estimation of leave-p-out. At least this is my understanding. – Fabio Oct 14 '15 at 13:38
  • 1
    Fabio, thanks. I'm leaving for a conference right now, and will look at the paper when I'm back. Meanwhile, the updated answer may help you to decide how many surrogate you need to test. I suspect that testing 2e6 models will not be necessary, as long before that you'll be in a region where model instability uncertainty << finite-no-of-independent-test-cases uncertainty. – cbeleites unhappy with SX Oct 14 '15 at 13:42