2

I've seen many questions posted about AUC but I'm still struggling to understand.

I see this definition for AUC everywhere "The AUC is an estimate of the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance."

What does the keyword "rank" mean?

Is AUC essentially a performance measure that measures how well a model can predict a binomial classification as being one class or another, whilst making minimal amount of classification error?

If that is pretty much the definition, then would that mean if your sensitivity and specificity are high (e.g. both are higher than say 80%) then AUC is likely to be high also?

Let's say I'm building a model that's trying to classify whether something is a cat or dog. Could someone explain the concept of AUC and relate it to this example?

Nick Cox
  • 48,377
  • 8
  • 110
  • 156
Lost Guy
  • 21
  • 1
  • 2
  • By *"rank...higher"* they mean that the probability that your model outputs will be higher for a randomly chosen positive instance than a randomly chosen negative instance AUC% of the time. – Dan Jan 04 '18 at 11:33
  • So my definition "Is AUC essentially a performance measure that measures how well a model can predict a binomial classification as either being one class or another, whilst making minimal amount of classification error?" is essentially correct then? – Lost Guy Jan 04 '18 at 11:34
  • 1
    Not really, it's being measured before the classification choice is made. It's used for comparing two models that model the probability of the outcome being in one class (as opposed to another class) which is not classification. Classification only happens after you choose a threshold (i.e. put everything with p > 0.6 in the positive class) – Dan Jan 04 '18 at 11:41
  • So in other words, AUC measures an estimate of how well a classifier model (e.g. decision tree, logistic regression), can predict a random sample (where actual label is the positive instance) as a positive instance in comparison to predicting a random sample (where actual label is the negative instance) as a negative instance. – Lost Guy Jan 04 '18 at 11:51
  • Say my positive instance is Cat and i'm using AUC to predict the performance measure of Decision Tree. If the AUC is 0.8, that means Decision Tree is estimated to predict random samples (where actual label value is cat) as a cat, better than if predicting random samples (where actual label value is a dog) as a dog. – Lost Guy Jan 04 '18 at 11:57
  • 1
    In that example it's saying that if you pick a cat and a dog at random and feed them both through your model, your model will assign a higher probability to the cat observation than to the dog observation 80% of the time. – Dan Jan 04 '18 at 12:26
  • Obligatory plug for the essay I wrote on this topic here: http://madrury.github.io/jekyll/update/statistics/2017/06/21/auc-proof.html – Matthew Drury Jan 04 '18 at 15:18

1 Answers1

1

The AUC is the area under the ROC curve (usually, sometimes the precision-recall curve is used, such as when there is class imbalance)

Consider this image by BOR at the English language Wikipedia, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=10714489

https://commons.wikimedia.org/w/index.php?curid=10714489

The curve is a plot of TPR vs FPR (or sensitivity vs 1-specificity). Note, your cat-dog classifier has only one value for both TPR and FPR and so is only a single point on this curve. However, the classifier is most likely actually a model that outputs the probability of an image being a cat and it only really becomes a classifier after you make a decision by thresholding that probability, e.g. classify as cat if $p > 0.6$. The ROC curve is generated by varying that threshold from 0 to 1.

If your threshold is 0, your classifier just decides all images are cats. That results in a TPR and FPR of 1 which is the top right corner of the chart. Conversely, a threshold of 1 mean no positive predictions (no cats) and thus a TPR and FPR of 0 which is the bottom left of the curve. The ROC curve always connects these two points. A model that is a random guess has an ROC curve that is the 45 degree diagonal, anything above this line (i.e. towards the top left) mean that the model is better than a random guess

If your sensitivity (TPR) is $0.8$ and your specificity is also $0.8$ (i.e. FPR of $0.2$) then you can see that your classifier is a point $(0.2,0.8)$ that is way above the diagonal $(0.2,0.2)$ (i.e. way better than a random guess). In fact a perfect classifier would be at $(0,1)$. But yes, a curve passing through $(0.2,0.8)$ is likely to also have a high AUC.

AUC is the area under the entire curve, not just a single point. This allows you to compare to models that model probability, not two classifiers. The choice of threshold gets made later and depends on your application. Are you more sensitive to precision or recall for example? In practice, you often won't even choose this threshold explicitly. For example, if you cat-dog classifier is used to profile animals at the Republic of Catopia international airport where they are looking to question dogs before allowing them on planes, they will only be able to question $n$ dogs an hour. So they might simply run the model and take the $n$ images with the highest probability. Now your threshold gets implicitly decided and can differ each hour. In this case, in order to choose the best model, you want a metric that measures across all threshold values and not just one. This is what the AUC is measuring. Note that is is just a summary statistic of the ROC curve and in the same way a mean doesn't tell you everything about a distribution, the AUC doesn't tell you everything about the ROC curve and so the curve itself is still a useful plot.

Going back to the plot above, the question is which model do you choose. which of those three curves is in general closest to the top left? It's difficult to say. the AUC is used as a simple measure to assist in this choice.

Dan
  • 1,288
  • 2
  • 12
  • 30
  • If one is interested in the $c$-index (concordance probability; area under ROC curve) the curve itself adds confusion to the problem instead of clarity IMHO. Stick to the original nice definition of concordance probability that you gave earlier @Dan. The $c$-index is an interpretable measure of pure predictive discrimination whereas the ROC curve is not that informative and is certainly not insight-giving. – Frank Harrell Jan 04 '18 at 12:57
  • Oh I completely get that definition now (The AUC is an estimate of the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance) by comparing the words "positive instance" with your y axis and "negative instance" along your x axis. – Lost Guy Jan 04 '18 at 13:49
  • @FrankHarrell what if you are comparing two curves both with the same AUC but one outperforms at high FPR and the other at low FPR. If your application is sensitive to FPR then couldn't the ROC curve be useful in selecting the right model for your task? – Dan Jan 04 '18 at 15:03
  • FPR is inconsistent with optimum decision making. See http://www.fharrell.com/2017/03/damage-caused-by-classification.html – Frank Harrell Jan 04 '18 at 16:28