0

I have a set of error rates and AUC values for classification methods, Logistic Regression, QDA, LDA, and KNN. It looks like:

Model         Error Rate           AUC
Logistic        .2533 (3)         .8001 (4) 
QDA             .2122 (1)         .8199 (3)
LDA             .2312 (2)         .8200 (2)
KNN             .2810 (4)         .8900 (1)

I ranked the models from (1)-(4) for which metric determined as the "best" fit. Clearly, as you can see the AUC values don't follow the error rates order from what should be the most effective model. Does anyone know how to interpret this?

ashman2222
  • 1
  • 1
  • To calculate an error rate, you need to convert the probability outputs to categories. This requires a threshold above which you round to $1$ and below which you round to $0$. How do you determine this threshold, software default? $\text{//}$ Even better for comparing models than AUC are the Brier score (mean squared error) and log loss. Both of these are strictly proper scoring rules. Among others on here, Frank Harrell and Stephan Kolassa are major supporters of using strictly proper scoring rules. – Dave Dec 18 '20 at 21:25
  • A few links about scoring rules: https://stats.stackexchange.com/questions/359909/is-accuracy-an-improper-scoring-rule-in-a-binary-classification-setting https://www.fharrell.com/post/class-damage/ https://www.fharrell.com/post/classification/ https://stats.stackexchange.com/questions/464636/proper-scoring-rule-when-there-is-a-decision-to-make-e-g-spam-vs-ham-email – Dave Dec 18 '20 at 21:26
  • I would definitely use the Brier score if given the option, but in this report, my professor wants the error rate output and the AUC score. So, I was just curious to if someone was familiar with why there is a difference between the two ranking results. – ashman2222 Dec 18 '20 at 21:35
  • Try it out with different classification thresholds. You will find that the error rate rankings shuffle around. Error rate is sensitive to the threshold. $\text{//}$ Why does she not want the Brier score? – Dave Dec 18 '20 at 21:42
  • Guess I'm just confused why KNN performed poorly on the error rate metric but performed well in the AUC metric – ashman2222 Dec 18 '20 at 21:43
  • Not sure why she doesn't want Brier score, I used it as an assessment for a win probability model I made and it worked great! – ashman2222 Dec 18 '20 at 21:43
  • The dataset might be imbalanced. – 4.Pi.n Dec 18 '20 at 23:22

0 Answers0