2

I have heard that the main reason to use out-of-bag sample(OBB) over CV is that OOB is faster to implement and usually gets the same results(for tuning the hyper-parameters) as implementing CV when the Random forest model is used.

If this isn't true, in which cases using CV gives better results?

J.Gastón T.I.
  • 23
  • 3
  • 1
    The question seems to be addressed in this post: [Out of Bag Error makes CV unnecessary in Random Forests?](https://stats.stackexchange.com/questions/207815/out-of-bag-error-makes-cv-unnecessary-in-random-forests?rq=1) – user20160 Jul 19 '17 at 06:41
  • 1
    Breiman also mentions this issue on his [random forest page](https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#ooberr) – user20160 Jul 19 '17 at 06:48

1 Answers1

2

Bootstrapping and CV have tradeoffs:

  • Bootstrapping can be used to generate an enormous number of "new" datasets which are "noisy" versions of the original dataset (in the sense that the samples are weighted differently than in the original dataset). However, each bootstrap sample sees approximately 2/3 of the samples, irrespective of the dataset size.

  • Cross validation allows controlling more precisely the number of samples used. In 10-fold cross validation, for example, each iteration will train on (ignoring roundoff errors) approximately 90% of the data.

Bootstrapping, therefore, is useful for things like training many high-variance low-bias predictors (like the trees in a random forest), or estimating confidence intervals of a statistic using many iterations. Conversely, it can underestimate the actual performance of the predictor.

As you pointed out in your excellent comment below, Breiman indeed states that the OOB error rate is a good estimator for OOS error rate. With all enormous respect due to Breiman, FWIW, I've seen different results, especially for datasets with very low SNR. Some of the answers to this question state this too. I assume that, asymptotically, for a given SNR, if the datasets grow large enough, Breimann is correct; in practice, you might want to be cautious about it.

Ami Tavory
  • 4,410
  • 11
  • 20
  • Regarding the speed issue...Bootstrapping is already needed to fit each individual tree. So, using the out-of-bag samples to estimate performance can be done 'for free', rather than re-fitting the forest k times for k-fold cross validation. – user20160 Jul 19 '17 at 06:58
  • @user20160 I don't think so. Bootstrapping is used to *fit* the tree using the OOB samples, and so *cannot* be used to estimate the performance. That would be an overestimation of the performance of the tree. Conversely, using CV for the forest, and *within each fold* using boosting, is OK. – Ami Tavory Jul 19 '17 at 07:05
  • 3
    Perhaps we're talking about different things? Individual trees are trained on in-bag points, whereas out-of-bag points can be used to estimate error. That's what Breiman himself recommends [here](https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#ooberr). Also mentioned in the chapter on RFs in Hastie&Tibshirani Elements of Statistical Learning. Of course, if using OOB points for hyperparameter tuning, an outer CV loop would be needed to estimate performance. – user20160 Jul 19 '17 at 09:03
  • @user20160 That is an excellent point (I assumed you meant that OOB is used to fit hyperparameters). Have updated the answer to address it. – Ami Tavory Jul 19 '17 at 09:15