1

It seems that the R packages I found around for computing the .632+ estimation of prediction error work only with categorical outcomes.

Why is that? Looking at the formulas in Efron 1997 paper it seems the estimator could be used with every kind of error.

Bakaburg
  • 2,293
  • 3
  • 21
  • 30

1 Answers1

2

As Frank Harrell notes in this answer:

You need modifications to the bootstrap (.632, .632+) only because the original research used a discontinuous improper scoring rule (proportion classified correctly). For other accuracy scores the ordinary optimism bootstrap tends to work fine.

Also, as discussed on this page, use of .632-type rules doesn't strictly follow a fundamental property of bootstrapping.

So I suppose that you could compute a .632+ score for other purposes, but there might not be much point. I suspect that accounts for any paucity of functions in R with respect to .632+ estimates.

EdM
  • 57,766
  • 7
  • 66
  • 187
  • Thank you. I'm aware of Harrell answer, but I found this article which uses .632+ on AUC showing that it is better the normal OOB bootstrap https://www.researchgate.net/publication/220913453_Comparison_of_Non-Parametric_Methods_for_Assessing_Classifier_Performance_in_Terms_of_ROC_Parameters. Now, I'm not sure what Harrel means with "discontinuous improper scoring rule" and if AUC is improper or not. In his post there is a link to his page course, not directly to a paper justifying his statement. – Bakaburg Nov 15 '19 at 17:45
  • @Bakaburg see [this page](https://en.wikipedia.org/wiki/Scoring_rule) about scoring rules, proper and improper. [This page](https://stats.stackexchange.com/q/339919/28500) discusses how AUC fits into that scheme (close, but not strictly proper). On an initial read of the paper linked in your comment, it doesn't seem to evaluate the ordinary optimism bootstrap (modeling on multiple bootstrapped samples, testing all models on full original data set) but rather a bootstrap-based evaluation that "test[s] on those cases that were not included in each bootstrap replicate" (page 5, bottom left). – EdM Nov 15 '19 at 18:22
  • thanks! I'll check them out! – Bakaburg Nov 15 '19 at 18:23