Does the local triangle inequality holds for Kullback-Leibler divergence

Question

Does the local triangle inequality holds for the Kullback-Leibler divergence? For the local triangle inequality, I mean the $$ d(\theta', \theta) + d(\theta'', \theta) \geq A d(\theta', \theta'') $$ for some $A \in (0,1]$. Also what about the square root of the K-L divergence?

Answer 1

The KL divergence isn't symmetric, so the answer might not be the same depending on how exactly you set up the "local triangle inequality"

For one set-up, the inequality can never be right:

$$ KL(p,q) + KL(p,r) \geq_? A KL(q,r)$$

because here is a counter-example: $p$ a gaussian centered at 0, $q_n$ a mixture of a gaussian centered at 0, and a gaussian centered at -n, with respective probability $\frac{n-1}{n}$ and $\frac{1}{n}$, and $r_n$ a similar mixture but with means 0 and n.

as $n \rightarrow \infty$, $KL(p,q_n) \rightarrow 0$ while $KL(q_n,r_n) \rightarrow \infty$

edit: I figured out a counter-example for the opposite direction

$$ KL(q,p) + KL(r,p) \geq_? AKL(q,r)$$

Once again no, because of another counter-example. Take $q_n$ a gaussian centered at -n, $r_n$ a gaussian centered at +n, and $p_n = 0.5q_n + 0.5r_n$

$KL(q_n,p_n)$ asymptotes to a finite value ($(\sqrt 2 - 1)H(g_0)$ if I'm not mistaken) while $KL(q_n,r_n) \rightarrow \infty$

open question: $KL(q,p) + KL(p,r) \geq_? A KL(q,r)$ ??