7

Score functions are used in the evaluation of probabilistic forecasts. The question: is the score function positive oriented or negative oriented?

For example, in the paper Strictly Proper Scoring Rules, Prediction, and Estimation it says that $S(Q,Q) \geq S(P,Q)$ for any distributions $P$ and $Q$,

Conversely, in the paper Predictive Model Assessment for Count Data it is the opposite, i.e. $S(Q,Q) \leq S(P,Q)$ for any distributions $P$ and $Q$.

What is the most common way to define a score?

plnalex
  • 73
  • 5
  • 4
    If you define a score function $S$ and I instead use $-S,$ both of us are performing identical analyses but with different conventions. So who cares? – whuber Oct 08 '21 at 13:28
  • 4
    @whuber: in theory, yes, of course you are right. In practice, when all you know is the "positive" convention and you read a paper that uses the "negative" convention, Much Confusion can result. Imagine growing up with temperatures measured in Fahrenheit and suddenly seeing a weather forecast in Celsius - but never having heard of this alternative way of measuring temperatures. – Stephan Kolassa Oct 08 '21 at 13:55
  • 1
    @Stephan That's why--as you remark in your answer (+1)--any decently written paper will lay out its conventions, if only implicitly by defining its score function. – whuber Oct 08 '21 at 14:00
  • @StephanKolassa In addition, I've rarely seen the word "score function" but I've commonly seen either of the words "cost function" or "reward function", and those are much more explicit about the convention: we want to minimize the cost, or maximize the reward. – Stef Oct 09 '21 at 07:05
  • 2
    @Stef: careful there! *Scoring rules*, especially *proper* and *strictly proper* scoring rules, have a very specific meaning in probabilistic forecast evaluation, namely that they map predictive *distributions* and outcomes to scores. I have never seen the more general "cost/reward function" with this specific meaning, whereas there are "cost functions" that are *not* scoring rules, like [accuracy](https://stats.stackexchange.com/a/359936/1352). I think being careful about nomenclature here can avoid a lot of confusion. – Stephan Kolassa Oct 09 '21 at 08:13

2 Answers2

8

Both "positively oriented" and "negatively oriented" scoring rules are common in the literature. Better papers explicitly note which convention they use.

Unfortunately, not everyone knows about the two common conventions. So people who think "their" convention is the "normal" one will not note which convention they are following - since after all, they can't imagine it not being shared by everyone. In such cases, you need to learn from the context which convention is being followed.

Stephan Kolassa
  • 95,027
  • 13
  • 197
  • 357
  • 4
    And thank you for reminding me. I just added a paragraph to the [scoring-rule tag wiki](https://stats.stackexchange.com/tags/scoring-rules/info) to draw attention to this potential confusion. – Stephan Kolassa Oct 08 '21 at 13:53
3

I think it’s useful here to refer back to wikipedia.

The first equation you quote above is for a proper scoring rule. This is one where the highest possible score is reported by using the true probability distribution.

I can’t access the second document referenced in the question so I can only guess that it is a) a typo or b) the score is defined such that negative values are optimal.

Adam Kells
  • 908
  • 1
  • 12