0

I am testing two predictive models, Each of them in 10 experiments.

I therefore have 10 ROC curves (with AUC) for first experiment and 10 ROC curves for second experiment.

How to compare these series and see if difference between them is statistically significant ?

How to get confidence intervals / P value ?

mdewey
  • 16,541
  • 22
  • 30
  • 57
suprvisr
  • 643
  • 2
  • 8
  • 14
  • Does this help with the confidence intervals https://stats.stackexchange.com/questions/204068/confidence-interval-of-auc-with-reitsma-model-in-mada-r-package/204549#204549? – mdewey Apr 28 '17 at 15:18
  • There is a r package that can do confidence intervals and p value for a given one series for one model. However I am trying to compare series of experiments for 2 different models.... – suprvisr Apr 28 '17 at 15:19
  • Do you have 20 independent experiments each giving you an AUC? – mdewey Apr 28 '17 at 15:38
  • No I have 10 for one model and 10 for another model - and need to compare these... – suprvisr Apr 28 '17 at 15:39
  • There is an R package metamisc but it does not allow for a moderator. You could e-mail the authors to see if he has any plans to include that or, since it uses metafor internally, you could hack it to include moderators. – mdewey Apr 28 '17 at 15:45
  • What do you mean by moderator? – suprvisr Apr 28 '17 at 15:46
  • @ suprvisr, is the target/dependent variable the same for each of the two predictive models? Also, you state: "I have 10 for one model and 10 for another model". Just re-confirming you are testing the first model on 10 experiments (10 datasets), and the second model on these same 10 datasets (not another set of 10 datasets)? – user3487564 Apr 28 '17 at 22:29
  • So it is AUC - empiric on cross validation. Randomly picking up 95% of dataset for - GBM and another model - training and testing both models on remaining 5% with each of 10 interactions. The target variables are the same. Yes. – suprvisr Apr 28 '17 at 22:49
  • I have deleted my answer as now you have explained what you did I do not think it is appropriate advice. – mdewey Apr 29 '17 at 08:12

0 Answers0