0

I have the following confusion matrix:

[[5584    0   11    6    6   24   27    3  262    0]
 [   0 6419   39   14    3   41    4    4  214    4]
 [  17   21 5287   67   59   19   48   34  402    4]
 [  20   13   95 5238    0  150   21   36  505   53]
 [   9   12   29    8 5245    6   26   13  384  110]
 [  21   15   21  127   45 4469   66   11  594   52]
 [  25   15   37    2   31   73 5566    6  163    0]
 [  18    9   43   13   37   12    4 5724  250  155]
 [  12   52   33   83    2  101   24    4 5509   31]
 [  19   18   25   51  102   25    1  147  517 5044]]

and I would like to calculate the accuracy of each class. I have found on the internet that it should be calculated as follows: (TP+TN)/(TP+TN+FP+FN) but when I try to implemented I basically get lost as I do not understand how to calculate any of the members except the TP which is the item on the diagonal.

I also thought to calculate it as the TP over the total number of times a class exists in the dataset (i.e. the row sum) but that would be equivalent to calculate the recall.

Would you be able to make a practical example on how to calculate the accuracy of class number 5 please?

This is what I tried so far in Python:

def accuracy_scores(confusion_matrix):
    msg = "accuracy\n\n"
    accuracies = []
    for i in range(confusion_matrix.shape[0]):
        tp = confusion_matrix[i][i]
        fn = sum(np.delete(confusion_matrix[i], i))
        fp = sum(np.delete(confusion_matrix[:, i], i))
        tn = sum(sum(np.delete(np.delete(confusion_matrix, i, 0), i, 1)))
        accuracy = (tp+tn)/(tp+  fn+  fp+  tn)
        accuracies.append(accuracy)
        msg += str(round(accuracy,2))+"\n"
    msg += f"accuracy avg: {round(np.mean(accuracies),2)}\n\n"
    return msg
Federico Gentile
  • 202
  • 1
  • 10
  • What exactly do you mean by the accuracy of a class? Using MNIST handwritten digits as an example, do you mean how many times you were presented with a picture of a $2$ and classified it as a $2$, how many times the classifications of $2$ turned out to be true images of $2$ (not the same as the previous), or something else? – Dave Jan 03 '22 at 17:00
  • The first one you suggested thanks – Federico Gentile Jan 03 '22 at 17:06
  • How many times were you presented with a $2?$ of those, how many were identified as $2?$ – Dave Jan 03 '22 at 17:07
  • How many times were you presented with a 2 – Federico Gentile Jan 03 '22 at 17:08
  • In MNIST test data, there are about $6000$ such instances. – Dave Jan 03 '22 at 17:09

1 Answers1

1

It's quite simple, using class number 5 as an example, just use take all the samples from class 5 as “positive” instance and all the other classes as “negative” instance, same for predictions. Just keep in mind that accuracy is not the best metric and can be misleading for unbalanced classes, what would be the case for multi-class classification data if you look only on the true vs not-true pairs.

Tim
  • 108,699
  • 20
  • 212
  • 390