draw roc curve on an example of 10 probability scores

Question

I'm studying machine learning and find an example question on the book which really confused me.

Q:

A scoring classifier is evaluated on a test set of 10 examples resulting in the following probability scores:

           0.9, 0.8, 0.7, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1

with true classes:

           1     1    0    1    1    0    1    1    0    0

The answer for the ROC curve is Could anyone give me some hints on how they draw this? I'm very confused on why the curve turn right when the tpr raise to approx 0.3 and 0.7. Thanks!

Answer 1

Create dataset of actual tag and score. Sort dataset by decreasing score. For every row, do the following (treat as rough pseudocode):

# calculate these
total.class.1.seen = # number of class-1 cases >= row.score
seen.cases = # number of cases seen till now.
total.class.1 = # total class 1 cases in data
total.class.0 = # total class 0 cases in data.

TPR = total.class.1.seen/total.class.1
FPR = (seen.cases - total.class.1.seen)/total.class.0

Now plot these TPR and FPR values. Essentially what you're doing at every row, is saying that anything which has score >= this score would be classified as class-1.

The above code doesn't do well when you have non-unique scores (As you have it for 0.7 here). Think of how you want to handle such cases. *Hint: for all unique scores, the plot would be a sequence of only vertical or horizontal line-segments, when non-unique, you get slant line segments.