Combine probabilities of several multi-class models

Question

Lets say I have two multi-class models (A and B) that are predicting whether or not a set of inputs belong to one of 5 classes. Each models predictions are probabilities that sum to 1. As an example, imagine the following:

+---------+---------+---------+--------+
|         | Model A | Model B | Result |
+---------+---------+---------+--------+
| Class 1 | 0.2     | 0.4     | ?      |
+---------+---------+---------+--------+
| Class 2 | 0.3     | 0.3     | ?      |
+---------+---------+---------+--------+
| Class 3 | 0.15    | 0.2     | ?      |
+---------+---------+---------+--------+
| Class 4 | 0.25    | 0.05    | ?      |
+---------+---------+---------+--------+
| Class 5 | 0.1     | 0.05    | ?      |
+---------+---------+---------+--------+

How would I go about combining these probabilities into one single probability that still sums to one?

Answer 1

The simplest approach would be to just take an average of the predictions for each class. You can use a weight.

Suppose the first model gives predictions $p_1, \dots, p_5$ and the second gives $q_1, \dots, q_5$. Then

$$\sum_{i=1}^5 p_i=\sum_{i=1}^5 q_i = 1.$$

Take any weight $0<w<1$, define the combined prediction by $r_i := wp_i+(1-w)q_i$. Then

$$ \sum_{i=1}^5 r_i = \sum_{i=1}^5\big(wp_i+(1-w)q_i\big) = w\sum_{i=1}^5 p_i+(1-w)\sum_{i=1}^5 q_i = w+(1-w) = 1. $$

So your predictions again sum up to 1. This also works for more than two classifiers.

As weights, you could use past performance of your two classifiers. Or take the unweighted average, $w=\frac{1}{2}$, which is frequently better than trying to estimate "optimal" weights (Claeskens et al., 2016, IJF).

Alternatively, you can use any other method to combine your predictions, and just renormalize the results afterwards to sum to 1.