1

I know there's been some discussion on differences between CV and bootstrapping for estimating out-of-sample prediction error of a classifier.

For example, in here (Differences between cross validation and bootstrapping to estimate the prediction error), here (Bootstrapping estimates of out-of-sample error) and here (What is the .632+ rule in bootstrapping?).

I'm interested in, however, maximizing AUC directly, and not the prediction error (1 - accuracy) itself as the cutoff points are not specified a priori.

Would the reasoning of the posts above apply? I find it difficult, for example, to calculate the AUC with only 10 observations (assuming a CV10 applied to a sample of 100 observations).

Currently I'm using the "optimism" bootstrap estimator, though it is pretty expensive (at least for the PC I access to).

Any thoughts?

Community
  • 1
Leonardo Cordeiro
  • 88
  • 8
  • See [this page](http://stats.stackexchange.com/questions/71184/cross-validation-or-bootstrapping-to-evaluate-classification-performance?rq=1) for further information, including some thoughts on why AUC might not be the best evaluation of your model. – EdM Jul 29 '15 at 19:18
  • AUC seems to be the more "neutral" evaluation metric as I don't have to specify a threshold (which is, in itself, a different problem altogether), required to calculate accuracy, for example. – Leonardo Cordeiro Aug 11 '15 at 19:31

1 Answers1

4

The AUC is equivalent to the c-index or concordance, the fraction of pairs of cases in which the ordering of the predictor value (based on the combination of predictor variables) is consistent with differences in outcome. So in principle if you wanted to do cross-validation you could use the paired comparisons of the held-out cases to calculate a concordance index (and thus an AUC) for each fold of CV.

But to get enough comparisons to be useful you would probably have to do not just one CV but multiple repeated cross validations with different subsetting of the cases. The many iterations of CV may offset your initial idea that fewer computing resources are required for CV than for bootstrapping. I find the bootstrap more straightforward and have used it for estimating standard errors of AUC values. If you only have on the order of 100 observations there shouldn't be that large a computational cost, so check the bootstrap algorithm you are using.

My hesitation is that although AUC might be considered a "neutral" evaluation metric, classifiers are typically used in a situation where there are different costs to false-positive and false-negative determinations. It's not clear that you would necessarily get the same result by "maximizing AUC directly" as you would by evaluating the cost-benefit tradeoffs in the context of how you plan to use your results. And your use of the word "maximizing" suggests that you might be trying to use AUC to compare among models, which might be better done with a different measure like the Akaike Information Criterion; see the accepted answer on this page and its comments.

Community
  • 1
EdM
  • 57,766
  • 7
  • 66
  • 187
  • For model selection I've run 100CV2, 50CV4, 40CV5, and 10CV10 (CV) to get 200 pairs for comparison each time. It is somewhat costly but nothing that a top Intel/AMD processor can't handle in a few minutes. I had to write critical parts of the code in C to make things run (way) faster, though. – Leonardo Cordeiro Aug 16 '15 at 01:08
  • Which flavor of bootstrap estimator do you use? Elsewhere Frank Harrell advocated the "optimism" boostrap but I can't seem to find a proper reference. Thoughts? I'm aware of what you mean about the AUC. I can't estimate error costs for this particular problem so I'm good with the whole AUC. AIC and BIC don't apply to my problem, I reckon. The number of parameters is fixed and there`s no likelihood function. – Leonardo Cordeiro Aug 16 '15 at 01:21
  • Try looking at the functions provided in Harrell's `rms` package in R, and at the notes on his academic website. – EdM Aug 16 '15 at 03:12
  • [This page](http://thestatsgeek.com/2014/10/04/adjusting-for-optimismoverfitting-in-measures-of-predictive-ability-using-bootstrapping/) is exactly on point. – EdM Aug 16 '15 at 03:18