4

I have to calculate the KL divergence between a distribution $q$ and a prior distribution $p$, both of which are univariate Gaussians, i.e. $KL(q|p), q \sim \mathcal{N}(\mu, \sigma^2), p \sim \mathcal{N}(\mu', \sigma'^2)$. This term is part of a larger formula, which is justified in some way not relevant to this question.

Now, to be honest, I don't want to put any asusmptions on $p$ except that it is Gaussian. My intuition is to just say that if I do that, I can just say $p=q$ and thus $KL(p|q) = 0$.

I wonder if there is a way to phrase this intuition into proper math. I read about non-informative priors, but found nothing about calculating KL divergences in this scenario.

Update:

I try to make the question more clear.

I have two distributions. One is the prior, the other driven by data. My prior is not tied to any parameter ranges (e.g. in form of a conjugate prior), but only specified in its functional form (i.e. disitrbution family). In typical Bayesian frameworks, such a thing is called an informative prior. Does the same concept exist for KL based objective functions?

bayerj
  • 12,735
  • 3
  • 35
  • 56
  • 2
    At the moment question is not clear to me. You have two univariate Gaussian distributions. What do you want to infer then? If you minimize KL between two univariate Gaussian distributions you really get $p = q$ and $KL(q|p) = 0$. Or you want to state some connection between prior and posterior distribution? What is the question you want to answer? – Alexey Zaytsev May 20 '14 at 07:33
  • 2
    You asked this question three years ago: http://stats.stackexchange.com/questions/7440/kl-divergence-between-two-univariate-gaussians – jaradniemi May 21 '14 at 20:44
  • The question is something else. I am interested in taxonomy only here. – bayerj May 26 '14 at 08:53
  • The update completely changes the question and is thoroughly confusing. A *conjugate prior* is always "specified in its functional form," so the distinction you make does not exist. – whuber May 26 '14 at 14:36
  • A non informative prior does not specify a belief on a parameter (e.g. $\mu = 0$) but only on the functional form (e.g. Gaussian). Usually, a prior also says something about the statistics, e.g. $\mu = 0$. – bayerj May 26 '14 at 18:01

1 Answers1

4

The answer to your previous question about this topic was

$$KL(p, q) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{2}$$

where you were using the notation $\sigma_1=\sigma, \sigma_2=\sigma^\prime$. To make the mildest possible assumption on $p$ you must take the supremum of $KL(p,q)$ as the parameters of $p$ range through all possible values; that is, as $\mu_2$ ranges through all real numbers and $\sigma_2$ ranges through all positive numbers. But note that this expression can be made arbitrarily large.

Rigorously, let $N \gg 0$ be any positive real number. Then by setting $\sigma_2 = \sigma_1 \exp(N+1/2)$,

$$KL(p,q) = \log\frac{\sigma_1\exp(N+ \frac{1}{2})}{\sigma_1} + \cdots - \frac{1}{2} = N + \cdots \gt N$$

where the omitted term "$\cdots$" is obviously positive. Therefore the supremum is $+\infty$.

Similar analysis (by finding the minimum of $0$ at $\sigma_2=\sigma_1$ and $\mu_2=\mu_1$ and observing that the divergence is a continuous function of its arguments $\sigma_2$ and $\mu_2$) shows that $KL(p,q)$ can attain all real values in the interval $[0,\infty)$. Without making any further assumptions on $p$, that is all that can be said.

Figure

Graph of $KL(p,q)$ for $p\sim \text{Normal}(\mu_2,\sigma_2)$ and $q\sim \text{Normal}(0,1)$. The vertical and standard deviation scales are logarithmic.

Community
  • 1
whuber
  • 281,159
  • 54
  • 637
  • 1,101