0

Should I calculate the classification performances (AUROC, AUPR etc) before or after the neural network calibration (using for ex. isotonic regression)?

andrea_a
  • 13
  • 3
  • It depends on the statistic. ROC AUC is a statistic of *ranks* so any calibration that does not change the order of the scores also won't change the value of the AUC statistic. On the other hand, other statistics work differently. – Sycorax Jun 03 '20 at 22:26
  • Thank you. If I use isotonic regression, would it change the ROC AUC? – andrea_a Jun 03 '20 at 22:27
  • Does isotonic regression change the order of the data points' scores? That is, is the isotonic regression that you're using non-decreasing or is it non-increasing? – Sycorax Jun 03 '20 at 22:28
  • It is non-decreasing – andrea_a Jun 03 '20 at 22:33
  • Then you have your answer: a non-decreasing function won't re-order its inputs! – Sycorax Jun 03 '20 at 22:35
  • What do you mean by saying that the ROC AUC depends on the order of the scores? – andrea_a Jun 03 '20 at 22:39
  • 1
    I mean that the scores themselves (0.15, 0.63, 0.99) don't matter, but the ordering of them (smallest, middle, largest) does matter for ROC AUC. See: https://stats.stackexchange.com/questions/105501/understanding-roc-curve/105577#105577 – Sycorax Jun 03 '20 at 23:09

0 Answers0