1

I am working on a 3-class classification problem. We are cross-validating via a Leave-One-Out Approach, and there are some instances where the test data has no instances of one of my three classes. For this reason, the corresponding row of my confusion matrix is a row of zeros.

I want a metric of how accurate my model is when predicting the other two classes. I would like to keep the third class in my model despite this occasional imbalance. However, the summary statistics that I have learned about in the past (Cohen's Kappa and the F1 score) fail when applied to these confusion matrices.

One way that I have tried to get around this problem is to delete the 2nd row of my confusion matrix. This would give a 2x3 confusion matrix which, while initially strange, does work within the formula of the F1 Score. I'm worried that this is not correct, however, and wanted to see what the community of StackExchange thinks. Thank you for your help!

Samuel Polk
  • 23
  • 2

1 Answers1

2

The best solution to your problem is to use different methods for evaluating your classification scheme.

First, it's best to use a proper scoring rule rather than things like F1 scores to evaluate a classification scheme. Your primary job is to model the probabilities of class membership; you can then use those probabilities and your evaluation of the costs of different mis-classifications to make whatever class-assignment decisions best meet the goals of your project. A proper scoring rule is optimized when you have modeled the true probability distribution. One example is the Brier score, related to the squared differences between actual class membership and predicted membership probabilities. The Brier score can be used for your 3-class problem. This issue is discussed extensively on this site; my answer here is one of many examples.

Second, leave-one-out cross validation is not a good choice. That point is made early on in this answer, which has extensive expert discussion of how to evaluate classification schemes.

Third, if membership in one class is of low prevalence, you might need to do your cross-validation or bootstrapping in ways that ensure examples of all 3 classes in all of your samples. For example, sample separately from each of the 3 classes at each fold, then put the samples from all 3 classes together for analysis.

EdM
  • 57,766
  • 7
  • 66
  • 187