3

It seems that similar arguments can be made for using nested cross-validation instead of a simple hold-out test set, as the arguments for using cross-validation instead of a single validation set. The main argument is that using cross-validation leads to a better approximation of out-of-sample model performance. The same should hold for using nested cross-validation instead of a single test set.

It seems that nested cross-validation would be particularly useful if you suspect there are covariate shifts in the data.

Is the main reason why nested cross-validation is not widely used, then, mostly computational, as it would require both significantly more computation time and careful thinking with regards to implementation? Am I correct that nested cross-validation should be strongly considered if you suspect there may be covariate shifts in the data, and so the performance on the hold-out test set may be a poor estimate for generalized out-of-sample performance of the model?

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
rinspy
  • 3,188
  • 10
  • 40

2 Answers2

1

To your first question as to why nested cross validation is not more widely used, what I've heard from people in industry who work with larger datasets (100Ks of samples) is that they don't even use cross validation, let alone nested cross validation, because it is too computationally expensive.

To your second question of when to consider nested cross validation, this answer makes sense to me. It says that nested cross validation is advised if a model selection process has high variance due to the data set size being small (which could lead to covariate shifts that you mention), and/or the model selection process being complex.

skeller88
  • 249
  • 1
  • 8
0

I have released a package that can help implementing nested cross validation in Python (for the moment, it only works for binary classifiers). If you want to check it out, it's here:

https://github.com/JaimeArboleda/nestedcvtraining

It's my first Python package, so any comments, suggestions or critics will be more than welcome!!

I post it as an answer because the parameters of the main function can be tuned so that you can perform either nested cross validation or only test+validation set using the skip_folds arguments.

Jaime Arboleda Castilla
  • 201
  • 1
  • 4