0

I know that we assess performance of classifiers typically with metrics like accuracy, ROC, etc. typically because we want to know whether or not a classifier can accurately predict an outcome. But, what if we are more interested in the probabilistic output of a classifier? As in, the quantity we care about is not the class that is predicted but the probability of that class. How can we go about assessing the performance of the model?

Peter
  • 1
  • 1
    It is perhaps worth looking into the Brier score https://en.wikipedia.org/wiki/Brier_score – Cam May 29 '18 at 01:57
  • 1
    There are metrics that measure how closely a probability aligns with a label. The most popular are the log-loss (used in logistic regression, classification trees, gradient boosting, and neural networks), and the brier score. Additionally, the AUC measures how well the probabilities put the observations in order, that is, how often a positive class is assigned a larger probability than a negative class. – Matthew Drury May 29 '18 at 04:08

1 Answers1

1

Look into proper scoring rules, including but not limited to the Brier score and log score. These scoring rules are used to measure the performance of probabilistic forecasts. A proper scoring rule is maximized when the predicted probabilities match the true probabilities. There also exists a taxonomy of score propriety, which includes "semi-proper" scoring rules. This Cross Validated answer provides a helpful description of the different types of score propriety.

Brash Equilibrium
  • 3,565
  • 1
  • 25
  • 43