multidimensional KL loss

Question

I read this question on Kullback Liebler Divergence

Now i'm have a multidimensional distributions, like these:

for example i try to predict if a person in image is a male:

sample(img)  |     P_true     |     P_pred
             |                |            
             | n/a  |no | yes |  n/a |  no | yes
 A           |  0   | 1 |  0  |  0.2 | 0.6 | 0.2 
 B           |  0   | 0 |  1  |  0.1 | 0.0 | 0.9
 C           |  1   | 0 |  0  |  0.4 | 0.2 | 0.4

where A,B,C are my examples, P_true is the groundtruth (the correct labels) and P_pred, n/a is the label that i can't say if is a male or not.

The p_pred are estimated probability vector (obtained by a softmax).

what is the formula for compute KL divergence with multidimensional probability vector?

Its the same formula as in univar case, just replace simple integral with multivariate integral — kjetil b halvorsen, Apr 11 '17 at 08:54
thus i have a kl value for each element of the array, correct? — sdrabb, Apr 11 '17 at 09:00
You must tell us much more, included how to read your arrays! But Kullback-Leibler is a distance between distributions, not between vales. — kjetil b halvorsen, Apr 11 '17 at 09:04

score 0 · Answer 1 · answered Apr 22 '17 at 09:49

This is one-dimensional case when you have $P_{true}$ and $P_{pred}$ distributions [each value in your table n/a, no, yes are random variables with assigned probabilities to them].

Let's consider $P_{true}$ as $P$ and $P_{pred}$ as $Q$ then $D_{KL}(P||Q)=\sum_ip_i \log \frac{p_i}{q_i}$.

There is a problem with computing $\log \frac{0}{0.2}$ so I'd recommend to replace all zeros with small values $\epsilon$.

Now you are able to calculate KL-divergence for each example A,B,C and even more :)

That's it.

An alternative could be found here: https://stats.stackexchange.com/questions/211175/kullback-leibler-divergence/248657#248657 — kjetil b halvorsen, Apr 22 '17 at 12:51

multidimensional KL loss

1 Answers1