17

Yesterday's question Determine accuracy of model which estimates probability of event got me curious about probability scoring.

The Brier score $$\frac{1}{N}\sum\limits _{i=1}^{N}(\text{prediction}_i - \text{reference}_i)^2$$ is a mean squared error measure. Does the analogous mean absolute error performance measure
$$\frac{1}{N}\sum\limits _{i=1}^{N}|\text{prediction}_i - \text{reference}_i|$$ have a name, too?

Richard Hardy
  • 54,375
  • 10
  • 95
  • 219
cbeleites unhappy with SX
  • 34,156
  • 3
  • 67
  • 133
  • Google allowed me to find [this paper](http://diskworld.wharton.upenn.edu/research/calibration_via_blackwell.pdf) where something very similar is named $L_1$-calibration score. Note that this score is a bit different than yours, anyway "$L_1$ score" seems the good keyword. – Elvis Jan 04 '12 at 17:23
  • What search terms did you use? Googling I mainly learned how many different tumour scores exist (L1 meaning lymphnode involvement in that context)... – cbeleites unhappy with SX Jan 05 '12 at 08:12
  • Something like "L1 score probability"... may be I've been lucky – Elvis Jan 05 '12 at 08:17
  • 1
    Or google tries to help me and thinks I'm looking for tumours because that's what I do more often... "probability near score L1" got me to the paper below. – cbeleites unhappy with SX Jan 05 '12 at 08:24

1 Answers1

21

Answer seems to be: no, because MAE doesn't lead to a proper scoring rule.

See Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications where the MAE is discussed under "Counterexamples of proper scoring rules".

cbeleites unhappy with SX
  • 34,156
  • 3
  • 67
  • 133