1

If I have 3 variables: Jimmy, Tommy and Jenny, and I trained 3 "one VS rest" models and got the following:

  • model accuracy for Jimmy = 99%
  • model accuracy for Tommy = 95%
  • model accuracy for Jenny = 70%

Will my combined model accuracy be $0.99 \cdot 0.95 \cdot 0.70$?

GarlicSTAT
  • 179
  • 1
  • 9

2 Answers2

3

Almost certainly not. Take any example, like 100 of each group. You correctly classified 99 instances of the first group, 95 of the second and 70 of the third group. Your overall accuracy is $$\frac{99+95+70}{100+100+100}=0.88\ \neq 0.66 = 0.99\times 0.95\times 0.70. $$ However, we do have $$ 0.88=\frac{0.99+0.95+0.70}{3}, $$ so a different hypothesis would be that overall accuracy is the average of the group accuracies.

And this does not necessarily hold, either. Suppose you have 100 Jimmies (99 correctly classified), 20 Tommies (19 correctly classified, and 10 Jennies (7 correctly classified). Your overall accuracy is $$\frac{99+19+7}{100+20+10}=0.96. $$

The correct combination is the weighted average, where each group's accuracy is weighted by the total group size.

(Also, accuracy is not very useful in the first place.)

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
0

The difference is almost into micro and macro averaging. You can look at this

A1010
  • 193
  • 8