1

I have a dataset with a non-binary target class $c$. I want to compute the AUC of my classifier and can do this easily using the one-vs-rest approach. I train $\binom{n}{2}$ classifiers where n is the number of different values that $c$ can take and compute the AUC for each of those classifiers then just take the mean of different AUCs.

The problem is that sometimes, the AUC is lower than $0.5$. This seems okay to me, since this applies for binary classes and my intuition tells me I should only be worried if the overall AUC was lower than $.25$ when the target class can take 4 different values. Is this logic flawed or is my intuition right?

I've also noticed that the AUC of some of the binary classifiers is lower than $.5$. In this case it should be okay to change it to $1-AUC$ since it's a binary classifier, or will this mess up the general result.

Any insights would be appreciated. Is my approach correct or am I messing up the overall score with my tampering?

Pavlin
  • 253
  • 2
  • 10
  • Possible duplicate of [How to plot ROC curves in multiclass classification?](http://stats.stackexchange.com/questions/2151/how-to-plot-roc-curves-in-multiclass-classification) – EdM Dec 12 '16 at 15:46
  • No. The question you have provided generally asks how to compute the AUC for multiclass problems. I know how to do this. My question is more theoretical as to what I can do with the actual pairwise AUC scores so I do not mess the overall score up. The question you have provided has very little to do with my question. – Pavlin Dec 12 '16 at 16:11
  • I'm not sure this is really a duplicate of the linked thread. This asks specifically if it is OK to use 1-AUC in the computation in place of the AUC. I don't see that addressed in the possible duplicate. FWIW, other relevant threads include: [Unbalanced dataset - ROC curve to compare classifiers?](http://stats.stackexchange.com/q/111478/7290), & [AUC for more than two groups?](http://stats.stackexchange.com/q/60545/7290) – gung - Reinstate Monica Dec 12 '16 at 16:23
  • 1
    You can't get an average <.5 a="" are="" aucs="" be="" better="" binary="" but="" classification="" coming="" component="" from="" generally="" get="" granted="" happened="" if="" in="" investigating="" is="" it="" might="" model="" multiclass="" of="" possible.="" problem.="" same="" see="" shouldn="" some="" that="" the="" these="" to="" unless="" what="" worth="" you=""> – gung - Reinstate Monica Dec 12 '16 at 16:26
  • @gung this was my thinking as well. But since it's already the binary classifiers giving me such a terrible AUC, there should be no harm in just flipping the output there, which would give me $1-AUC$ score. I'm just trying to verify this wouldn't somehow mess up my overall score. – Pavlin Dec 12 '16 at 17:16
  • I don't really know, so someone else will have to answer, but I strongly suspect that you should not use 1-AUC. – gung - Reinstate Monica Dec 12 '16 at 17:25

1 Answers1

0

I have found out that I cannot change the AUCs in any way. This would be incorrect since this skews the overall score. If I were to do this, I would have to know in advance (or have a method to determine) which of classifiers would have to be flipped. Doing so after the fact is not okay.

Pavlin
  • 253
  • 2
  • 10