27

Most resources on proper scoring rules mention a number of different scoring rules like log-loss, Brier score or spherical scoring. However, they often don't give much guidance on the differences between them. (Exhibit A: Wikipedia.)

Picking the model that maximizes the logarithmic score corresponds to picking the maximum-likelihood model, which seems like a good argument for using logarithmic scoring. Are there similar justifications for Brier or spherical scoring, or other scoring rules? Why would someone use one of these rather than logarithmic scoring?

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
Ben Kuhn
  • 5,373
  • 1
  • 16
  • 27
  • 2
    Some hints are in the nomenclature. "Cost functional" is from optimization or optimal control system engineering. There is no "best". To have a "good" means you must have a measure of goodness. There are infinite numbers of families of measures of goodness. A trivial example is: what is the best path? If you are marching to your execution - make it a long pleasant one. If you are going to your Fields metal, make it shortest. System expertise helps you select the measure of goodness. When you have the measure of goodness, then you can find "best". – EngrStudent Dec 12 '14 at 19:05
  • 1
    You may be interested in [Merkle & Steyvers, "Choosing a strictly proper scoring rule" (2013, *Decision Analysis*)](https://doi.org/10.1287/deca.2013.0280). – Stephan Kolassa Mar 22 '18 at 16:22
  • 2
    I took the liberty of editing the title to make it more precise/informative. If I misinterpreted it, sorry and feel free to revert back the change. – Richard Hardy Sep 19 '19 at 11:30

1 Answers1

20

Why would someone use one of these rather than logarithmic scoring?

So ideally, we always distinguish fitting a model from making a decision. In Bayesian methodology, model scoring & selection should always be done using the marginal likelihood. You then use the model to make probabilistic predictions, and your loss function tells you how to act on those predictions.

Unfortunately in the real world, computational performance often dictates that we conflate the model-selection and the decision-making and so use a loss function to fit our models. This is where subjectivity in model selection creeps in, because you've got to guess just how much different kinds of mistake will cost you. The classic example is a diagnostic for cancer: overestimating someone's probability of cancer is not good, but underestimating it is much worse.

As an aside, if you're looking for guidance on how to pick a scoring rule, you might also want to look for guidance on picking a loss function or designing a utility function, as I think the literature on those two topics is a lot more voluminous.

Alberto Santini
  • 103
  • 4
Andy Jones
  • 2,146
  • 9
  • 10
  • 4
    1) Are you saying that Brier scoring is essentially a "loss function in disguise"--that is, even though it masquerades as a utility-function-agnostic scoring/comparison rule, it's actually used because people have specific preferences over the types of errors the model makes? – Ben Kuhn Dec 15 '14 at 18:54
  • 2) Do you have any specific examples of settings in which someone might choose Brier or spherical scoring over log scoring (= marginal likelihood, as I understand it) for those reasons? – Ben Kuhn Dec 15 '14 at 18:55
  • 3) Why would it perform better to bake your loss/utility function assumptions into the model than to fit to marginal likelihood and use your loss/utility function when actually making the decision? It seems like for ideal learning algorithms there should be no gap between these. – Ben Kuhn Dec 15 '14 at 19:01
  • 5
    1) Yep. 2) Not personally, no. Scoring rules aren't "fashionable" in the bit of ML I work in. Having a quick poke around on [Scholar](http://scholar.google.co.uk/scholar?q=spherical+scoring+rule&btnG=&hl=en&as_sdt=0%2C5), it seems they're a bit dated in general. [This paper](http://faculty.engr.utexas.edu/bickel/Papers/QSL_Comparison.pdf) looks like it'd be interesting to you though. 3) By performance I meant "computational performance", not "predictive performance". – Andy Jones Dec 15 '14 at 19:45