4

I have data in which groups of experts make proportion estimates.

I've been encouraged to use the ALRE method of scoring the error of these estimates. I found an article which describes this method:

enter image description here

As I understand it, the purpose of the ALRE method is to obtain a measure of error that is not dominated by single predictions that are wildly wrong. For example, answers to a question like "What is the distance (in km) between the Mars and Neptune?" could be wildly wrong.

However, all my data is bounded by 0 and 1 and thus I'm not sure if this measure is appropriate for me. If this is not a suitable measure for my sort of data, what should I use?

The observed values are 1s and 0s in some of my data sets. In some of my other data sets the observed values are also proportions.

user1205901 - Reinstate Monica
  • 11,303
  • 26
  • 77
  • 152
  • 1
    Are the observed values that you have 1s or 0s? Or are they proportions too? I don't know this ALRE but if you have a predicted probability distribution, any proper scoring rule should work fine. See e.g. my answer here http://stats.stackexchange.com/questions/71720/error-metrics-for-cross-validating-poisson-models/71936#71936 – Momo Jan 10 '15 at 00:24
  • The observed values are 1s and 0s in some of my data sets. In some of my other data sets the observed values are also proportions. This is certainly something I should have mentioned in the OP, so I have edited it. Could you explain a little further what you mean by any proper scoring rule being fine if I have a predicted probability distribution? – user1205901 - Reinstate Monica Jan 10 '15 at 00:34

0 Answers0