3

I am using a neural network with 5 input neurons, 2 hidden layers of about 50 neurons in each layer, and 4 output neurons, trying to classify my 5-dimensional data into 4 different classes.

Currently, I am choosing the network classification for a given example as the that corresponding to the neuron with the highest output and as expected, these results have been less than good.

How would I score my multiclassifier and subsequently choose different thresholds to improve my results?

So far, what I have tried is creating ROC curves by considering each class alone and the other 3 as a negative class, using a simple scoring method of fp/(1-fp) and choosing different thresholds for calculating the confusion matrix. How would I choose a score and thresholds that consider all classes rather than just 1 and the negative?

Thanks, Annika

annikam
  • 53
  • 6

1 Answers1

2

Train your network to give you probabilistic predictions $\hat{p}_{i1}, \dots, \hat{p}_{i4}$ for membership in your four classes of the $i$-th instance.

Then you can use proper ("scoring rules" in the technical sense; I can't tell whether you are using the term in this sense - take a look at the tag wiki for more info) to assess these. For instance, if the $i$-th instance turns out to belong to class $j(i)$, then the logarithmic score would penalize the corresponding predictions by $-\log(\hat{p}_{i,j(i)})$, and the Brier score would penalize it by $\hat{p}_{i,j(i)}^2$. In each case, smaller is better. You would average these scores over multiple instances $i$ and prefer models with lower average scores.

Once you have well-calibrated probabilistic classifications, you can tune your thresholds for optimality, which will involve not only the predictions, but also the costs of misclassification. See here for more info. That is, you should separate the prediction and the decision aspects of your problem.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357