2

I have an algorithm which outputs a confidence score and a probability score that a particular user belongs to class $C_i$, for multiple values of $i$. I want to output a single class $C_o$ as the final prediction. How do I combine these two independent metrics to output a single prediction?

Ankit
  • 21
  • 1

1 Answers1

1

Your prediction exercise ended when you output the confidence and probability scores. When you decide whether to treat a particular user as belonging to a certain class, you are making a decision. The probabilistic output of your model is one input to the decision, but others are at least as important. For instance: how costly is it to treat a user as a member of $C_i$ when he is actually a member of $C_j$ (and vice versa - note that the costs may be asymmetrical)? Some more information here.

So: try to quantify the costs of misclassifications. Given your model and these costs, it should be easy to find the class that minimizes expected costs, or optimizes any other objective function you are interested in.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • If it isn't possible to quantify the cost of misclassification, what approach could I follow to choose a class? The approach I'm thinking of is considering those classes whose confidence score is greater than the median of all confidences and picking the one with maximum probability. Can we do something more elegant than this? – Ankit Dec 19 '18 at 12:28
  • 1
    I very much doubt that it is not possible to assign a cost to misclassification. (If it is, then just assign everything to a fixed class and take the rest of the day off. If this is not acceptable, this implies that there is a cost to the misclassification, So let's quantify it.) Yes, it may be hard. To a first approximation, you can simply assign equal costs to every misclassification and see where this leads you. Your proposed approach amounts to picking one very particular cost structure. It may be instructive to try to understand your approach's implicit costs. – Stephan Kolassa Dec 19 '18 at 13:11