3

Say $X$ is a discrete random variable with cardinality $|X|$ and $Y$ is a discrete random variable with cardinality $|Y|$.

Does it make sense to talk about the KL divergences $D_{KL}(X||Y)$ or $D_{KL}(Y||X)$ of these 2 probability distributions if $|X| \neq |Y|$ ? If so, how does one compute it ?

locke14
  • 175
  • 6
  • 7
    From the [definition](http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Definition): "The K-L divergence is only defined if $P$ and $Q$ both sum to $1$ and if $Q(i)=0$ implies $P(i)=0$ for all $i$ (absolute continuity)." Therefore the answer in general is: no. –  Oct 15 '12 at 09:26
  • 4
    [A related question](http://stats.stackexchange.com/questions/6907/an-adaptation-of-the-kullback-leibler-distance). Didier's answer there, in particular, is highly relevant. – cardinal Oct 15 '12 at 10:07
  • 1
    An example: If $X$ has the possible values $0,1,2$ and $Y$ has the possible values $0,1,2,3$ and your observation is $3$, then $X$ is excluded as a model by usual, determinisic logic! Sothe power to discriminate between these twomodels is infinitely large, and you could , if you want, define KL-divergence from $X$ to$Y$ to be $\infty$, if that makes sense. The KL-divergence in the other direction could be finite. – kjetil b halvorsen Oct 15 '12 at 14:46
  • 1
    Yes, it makes sense to think of $D_{KL}(Y||X) = \inf$ and compute $D_{KL}(X||Y)$ for values $0,1,2$ in your example. – locke14 Oct 17 '12 at 11:19

0 Answers0