2

I want to compare a given classification algorithm with others via the Area under the (ROC-)curve metric. Unfortunately this algorithm only outputs the values of the respective confusion matrix (TP, FP, TN, FN) and a subset of the predicted positives, but no probability score for any of its predictions.

Confusion Matrix and Statistics
      Reference
Prediction TRUE FALSE
 TRUE    11    10
 FALSE    3   475

           Accuracy : 0.9739          
             95% CI : (0.9559, 0.9861)
No Information Rate : 0.9719          
P-Value [Acc > NIR] : 0.46294         

              Kappa : 0.6156          
 Mcnemar's Test P-Value : 0.09609         

         Sensitivity : 0.78571         
         Specificity : 0.97938         
      Pos Pred Value : 0.52381         
      Neg Pred Value : 0.99372         
          Prevalence : 0.02806         
      Detection Rate : 0.02204         
Detection Prevalence : 0.04208         
   Balanced Accuracy : 0.88255

When I tried to understand the ROC with examples like this or this, it always requires the prediction score to calculate the AUC and draw the curve. Wikipedia hints, that I should use a probability density function, but I don't know which and how. So, is it even possible to calculate the score and if yes, how?

Thank you guys in advance for your replies.

Hendrik
  • 143
  • 4
  • You need some sort of quantitative score if you want to make a ROC curve that has more than just one point. If you can show how got this data maybe someone can help you extract the numerical predictions from the model. – Calimo May 07 '18 at 16:43
  • Some R classifiers will output probabilities, but since you do not say which one you are using, we can't help you find that. – G5W May 07 '18 at 19:21
  • I'm afraid it is not a standard R classifier, but one I implemented out of a research paper. I found out there may be a way to calculate the scoring values. If so, I will write a short answer. – Hendrik May 08 '18 at 06:18

0 Answers0