Do you perform cross validation on bootstrap samples?

Question

Let's say that I have to chose the best model to predict a variable but my sample size is small. I would like to resample my data using the bootstrap, run each model and evaluate its prediction error with cross validation on each bootstrap sample. In the end I will chose the model with the lowest average prediction error.

Is this a acceptable procedure? If yes is there something I should be aware of while using this procedure?

score 1 · Answer 1 · answered Aug 22 '12 at 10:41

1

The way the bootstrap is applied to estimate prediction error rate is to use the bootstrap samples to estimate the bias of the resubstitution estimate of error rate and adjust it. Various bootstrap estimators have been devised and shown to work better than the leave-one-out version of cross-validation. For most population distributions the version called 632+ is the best. Details about the methods and the literature on it can be found in my book Bootstrap Methods or this book by McLachlan. This started with Efron 1983 JASA paper "Estimating the error rate of a prediction rule: Improvements on Cross-validation ".

This approach is preferable to the method you have planned.

answered Aug 22 '12 at 10:41

Michael R. Chernick

39,640
28
74
143

I have already found something on the web something referring to the boostrap estimators 632+ but unfortunately my mathematical background doesn't allow me to understand it clearly from the papers. Is there a way to explain it easly ? And subject to the fact that bootstrap estimator 632 is a better approach i would like to know which are the problems that affect my solution? – frin Aug 22 '12 at 14:07
@frin Exactly why 632 works so well is not so easy to understand. I try to explain it by saying that it gives the appropriate coefficients to a linear combination of two estimated to keep the variance and bias both small. It keeps the bias small because it weights the apparent error rate and e0 in a way that nearly cancels their opposing biases. Understanding 632 requires soMe knowledge of the bootstrap e0 estimate and the apparent error rate (the former is a little diffucult to undersatnd and the latter rather easy). – Michael R. Chernick Aug 22 '12 at 15:54
I'd have said that a bit of optimistically biased resubstitution is mixed into the independent out-of-boostrap estimate in order to hopefully correct the pessimistic bias... As I deal with data where the resubstitution is often heavily overoptimistic (e.g. underestimating the true error by factor 3 - >10), I directly go for out-of-bootstrap. – cbeleites unhappy with SX Aug 22 '12 at 15:59

score 1 · Answer 2 · edited Apr 13 '17 at 12:44

is there something I should be aware of while using this procedure?

You should be aware of the fact that prediction error estimates are subject to bias and variance. With small sample sizes, the variance is the problem.
To get a first idea of the variance of your prediction error estimates, look at the variance of prediction error within and between bootstrapped models that are built using the same input variables. While this can underestimate the true variance, it will give you a first idea of what is going on.
For this, you can e.g. look at the model built using all available input variables, or a model built with input variables that you as an expert for the application think will be a good choice.
Next, comparing prediction errors is basically a statistical test. Fortunately, you can use paired tests if all competing (surrogate) models were tested using the same test samples. Unfortunately, comparing many models means that you have a massive problem of multiple comparison.
From the results of the variance estimation above, calculate how much better a model needs to be so you can recognize the improvement. Is that something that you can realistically expect? If not,
- you either need larger sample sizes or
- you better stick to the first model and report that the available sample size does not allow any further optimization.
Whenever you do data-driven model optimization$^*$ you need to test the final model with a new independent test data set.
In other words, you need to apply double or nested cross validation (or out-of-bootstrap).
$^*$i.e. tune something [here: which variables to include] by test results.

You may also want to have a look at a similar recent discussion: Grid search on k-fold cross validation

Do you perform cross validation on bootstrap samples?

2 Answers2