1

I am familiar with the concept of combining forecasts. In the simplest scenario, that could be just averaging the forecasts. For example, let's say I have three forecasts representing the likelihood of an event such as (35, 50, 40). The combined forecast would be (35+50+40)/3 = 41.67 (although I realize there are some more sophisticated methods). This seems straightforward enough. Although I understand there are some risks associated with this, I was under the impression that combining forecasts is a common approach as in this question's answers.

However, let's say I have a situation that has multiple forecasts for multiple different groups (alternative options). As an intuitive example, a list of orders to be picked up from different locations.

order  prob  location
1      0.4   A
2      0.5   A
3      0.35  A
4      0.95  A
5      0.6   B
6      0.5   B
7      0.75  B 
8      0.15  B

Each order has a probability of being accepted (i.e. forecast). I want to say the probability of which location an order will be accepted. My initial instinct was to average the probabilities but I'm hesitant.

Note - the number of probabilities could exceed 100 per group and there could be >20 groups.

cdeterman
  • 4,543
  • 1
  • 20
  • 34

1 Answers1

0

The combined forecast would be (35+50+40)/3 = 41.67

This is known as Simpson's paradox, a very dangerous approach to share values. The items you work with are probability estimations derived on samples of different sizes, and by averaging the share you will get a confusing result if the actual numbers of right / wrong answers are really different across groups.

It is better to make counts of right answers and divide them by total counts.

Alexey Burnakov
  • 2,469
  • 11
  • 23
  • The intent is to use the results as a forecast, so the 'right' answers are not known. – cdeterman Oct 05 '17 at 15:17
  • OK, so I will clarify now what I meant. Averaging probabilites is a popular practice, as you would do when creating a forest of decision trees with binary output. I wanted instead to suggest using exact counts if you accumulated them. If those counts come from samples of different size, then instead of averaging probabilites (shares), you can divide "success" count (as you would do with a number of successes accumulated in terminal tree node) by total count for let's say location A. – Alexey Burnakov Oct 05 '17 at 15:43
  • So you are simply suggesting a weighted average by group size? – cdeterman Oct 05 '17 at 16:41
  • Yes, the weighted average according to a train sample size from which you derived the probabilities. – Alexey Burnakov Oct 05 '17 at 16:43