For practical reasons, I want to implement a standard confidence level which represents the probability that a particular instance is in a particular class given a model and the accuracy of that model on a number of examples. Sounds like something that is generally useful for machine learning applications but when I search for this answers are about the confidence level of the model itself, not the particular example.
Example: Model A gives a score of .3 of apple being a fruit. If we don't know whether apple is a fruit, how confident are we that apple is a fruit given the score of .3? Well, we also know that the model gave these scores earlier: .35 to orange, which is a fruit (correct) .30 to potato, which isn't a fruit (incorrect) .2 to banana, which is a fruit (correct)
So when we get a score of .3 from model A, with what probability is it a fruit?
Here's a try:
We can think of it as classifying whether the model is correct in this case or incorrect. Let's try Bayes Rule:
- The probability of model A being correct given score .3 = P(A|.3) = (P(.3|A) * P(.3)) / P(A).
- P(.3|A) = probability of .3 using the mean and standard deviation of all model A scores.
- P(.3) = the probability of .3 given all the other scores that the model has given
- P(A) = the probability of model A ever being correct with score .3 = number of fruit with a score above .3 / number of non-fruit with a score over .3 = 1/1 (set to 1 if denominator is 0) = 1
Question: Is the above approach correct, or is there something else key to understand about our confidence level that a model score indicates true or false for a class or label being true?
Alternative solution:
- The probability of class X being correct given a class X score of .3 by model A = P(X|.3,A) = (P(.3|A) * P(.3)) / P(A).
