2

For AUC ROC, why do we have a graph of multiple thresholds when in the end, we will only use one of those thresholds (so why not just choose the threshold and compare that one value across models)?

ramen_noodles
  • 71
  • 4
  • 2
    How would you choose a threshold if you only evaluate your models at a single threshold? Additionally, it's not generally true in professional settings that "we will only use one of those thresholds". Sometimes we care about risk/profit estimation, or ranking, or we want to let some other team manage threshold setting while we focus on the machine learning/analysis bit. The single pre-set threshold thing mostly only shows up in toy problems and beginner educational material. – Matthew Drury Apr 01 '21 at 22:56
  • 1
    You’re (probably) not doing a classification but a probability prediction. Please refer to Frank Harrell’s blog. https://www.fharrell.com/post/class-damage/ https://www.fharrell.com/post/classification/ – Dave Apr 01 '21 at 23:02

1 Answers1

2

ROC AUC is defined as the area under the ROC curve. The reason that ROC AUC considers multiple thresholds is that the ROC curve does. The reason the ROC curve compares multiple thresholds is to show the tradeoffs for true positive rate and false positive rate as the threshold is varied. Indeed, this is the original application of ROC curves.

Another reason that comparing multiple thresholds is useful is that it proportional to the Mann-Whitney $U$ statistic, which expresses the probability that a random element of one class is ranked higher than a random element from a second class. This second interpretation gives us some intuition about how to understand ROC AUC.

Alternatively, if you don't need a threshold to solve your problem, then you might consider a different statistic such as the Brier score for comparing models.

Sycorax
  • 76,417
  • 20
  • 189
  • 313