0

I got confused over a simple concept. Imagine that I have a binary classifier with 50% accuracy. So, if there are 10 samples to be classified as "y", "n", it has predicted 5 of them correctly.

Now, Imagine that I just guess the categories for each sample randomly (50% chance of getting it right for each one).

For the second one, it should be:

P(5 success) = 10!/(5!*5!) ((0.5)^5 * (0.5)^5) = 0.246

Are these two comparable?

This just came out of a comment from a friend who said "your model is just as good as a random guess", and I want to prove him wrong, but not sure how to put it together.

towi_parallelism
  • 183
  • 11

3 Answers3

2

I agree with wrtsvkrfm in the sense that accuracy is not helpful. I disagree, though, in that I don't think sensitivity and specificity are any more helpful. I give my arguments here.

Specifically, the more useful comparison of your classifier in terms of accuracy is not to a random classifier. Instead it is to the best naive classifier, i.e., the one that yields the highest accuracy. And that is not the random one. It is the one that always classifies as the majority class. For instance, if class A occurs 60% of the time and class B 40% of the time, classify everything as A to immediately have an accuracy of 60%. If your classifier achieves only 50%, then you are worse than this. This is one reason why accuracy is particularly problematic if you have unbalanced classes, but as above, I argue that it is not the right tool even for balanced ones.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • Thanks, I read all the comments. Still not sure how to look at it. Let's put it this way. Is there any way to find out the probability of my model achieving 50% success? I just think I am comparing orange and apple. 50% accuracy is not the same as probability of 50% success. how do I use a single metric to compare the two approaches? – towi_parallelism Sep 20 '18 at 09:21
  • 1
    Unfortunately, there is no simple good way to assess classifiers. Accuracy is simple, but not good. [Tag:scoring-rules] are good, but not simple. I don't fully understand what you mean be "the probability of achieving 50% success". You may be talking about [the TPR](https://en.wikipedia.org/wiki/Sensitivity_and_specificity), which has the exact same problems as accuracy. – Stephan Kolassa Sep 20 '18 at 09:50
0

When you have $n$ classes, and your classifier's accuracy is nearly $\frac{1}{n}$, it is customary to say that "your classifier is as good as random classifier"; because a random classifier would predict approximately $\frac{N}{n}$ instances correctly as yours, if you have $N$ samples (ignoring class imbalance situations for the simplicity of the discussion). So, he has a point.

Besides, your approach has flaws. We are concerned with the expected success, not the exact probability of having $\frac{N}{2}$ correct classifications. Imagine, your number of samples increases up to $N = 100$, then $P(50 \ \ \text{success})$ would be approximately $0.08$. Here, you're ignoring the neighbourhood.

gunes
  • 49,700
  • 3
  • 39
  • 75
0

It's not the accuracy that counts, but rather the sens, and spec. Take, for example, a sample size of $n=100$, with $n=5$ diseased and $n=95$ non-diseased. Assume further that the classifier is a shoe box which does nothing, then the classifier will spit out 100 votes for nothing (non-diseased). In this case it's accuracy will be 95% since only 5 objects will be truly misclassified. However, when running sens and spec, they will pick up the failure.

In other words, don't get hung up on accuracy.