1

I am struggling with choosing metric that I will use to compare models performance and hiperparam search. My task is similar to fraud detection.

I have found out that many people states PR is better curve than ROC for comparing performance on imbalanced data (absolutely my case). I will run probably run hundreds of experiments so I would rather stick to one metric with one number.

That is why I am looking for AUCs of PR and ROC. I found out that imbalanced data has actually bigger influence on PR AUC than ROC AUC. Do you think I should use ROC AUC or FPR at given recall point or there is something wrong with my plots? Is PR curve better tha ROC curve for imbalanced data but it is opposite with their AUCs?

Scores obtained by sampling from different Gaussian dist with mean = 0.1 for class 1 and mean=1 for class 1. Scores Histogram: green- scores for class 1, blue scores for class 0

enter image description here

enter image description here

jacekblaz
  • 31
  • 2
  • I don't have any authoritative answer on this, but my understanding is that the PR-curve should be preferred over the ROC-curve as the latter suffers from a bias: it can correctly predict a large number of true negatives at different thresholds, for unbalanced data, without incurring a penalty. Maybe this paper is of any assistance: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432 – horseoftheyear Dec 17 '19 at 19:51
  • 2
    Does this answer your question? [ROC vs Precision-recall curves on imbalanced dataset](https://stats.stackexchange.com/questions/262616/roc-vs-precision-recall-curves-on-imbalanced-dataset) – Calimo Dec 17 '19 at 21:32
  • @Calimo I have seen this thread dozens time while searching for answer. The thing is that it is about curves. That is what I am pointing on- my experiment shows that PR AUC is strongly influenced when imbalance in data is introduced. I do not understand why this is behavior I am looking for and thread you are posting does not explain that in my opinion. – jacekblaz Dec 18 '19 at 10:21
  • See also https://stats.stackexchange.com/q/7207/36682. If that doesn't answer your question then you need to rewrite it to make it clear what you're asking. – Calimo Dec 18 '19 at 15:31
  • One tmore hing that might help. When you write: "I found out that imbalanced data has actually bigger influence on PR AUC than ROC AUC", that's exactly what people mean when they say that ROC is insensitive to class imbalance. Maybe re-read the other questions with that in mind. They do answer all the questions you have here. – Calimo Dec 19 '19 at 07:24
  • @Calimo Thank you for patience. Now I think my question is: how being sensitive to imbalance makes PR AUC better metrics for imbalanced problem? Or how makes it better metrics for "needle-in-haystack type problems"? – jacekblaz Dec 19 '19 at 12:54
  • 1
    This is somewhat addressed already in the second link I posted. The accepted answer is spot-on answering the question. And rightly concludes "IMHO when writing a paper you should provide whichever curve answers the question you want answered [...]", hinting that it is not "just" better for needle-in-haystack. – Calimo Dec 20 '19 at 07:42

0 Answers0