12

Let $f$, $g$ and $h$ be densities and suppose you have $x_i \sim h$, $i \in \mathbb{N}$. What happens to the likelihood ratio $$ \prod_{i=1}^n \frac{f(x_i)}{g(x_i)} $$ as $n \rightarrow \infty$ ? (Does it converge? To what?)

For instance, we may assume $h = g$. The general case is also of interest.

Olivier
  • 738
  • 5
  • 22
  • Possible duplicate of [Kullback-Leibler divergence WITHOUT information theory](http://stats.stackexchange.com/questions/246239/kullback-leibler-divergence-without-information-theory) – Xi'an Jan 04 '17 at 06:34
  • 4
    @Xi'an. I think adding this question to SE allows for the connection to be drawn across questions in the answer. While there may be answer similarities the questions are not he same. – John Jan 04 '17 at 06:57
  • 1
    Thanks for the link. The question is not a duplicate, even though answers to my question *may* involve the Kullback-Leibler divergence. – Olivier Jan 04 '17 at 14:57

2 Answers2

16

If one takes the logarithm of this product, $${\mathfrak{r}}=\log \prod_{i=1}^n \frac{f(x_i)}{g(x_i)} = \sum_{i=1}^n \log\frac{f(x_i)}{g(x_i)}$$and turns it into an average $$\bar{\mathfrak{r}}_n=\frac{1}{n}\sum_{i=1}^n \log\frac{f(x_i)}{g(x_i)}$$the law of large numbers applies, hence one gets the almost sure convergence $$\bar{\mathfrak{r}}_n\stackrel{\text{a.s.}}{\longrightarrow}\mathbb{E}_h\left[\log \frac{f(X)}{g(X)}\right]=\int_\mathfrak{X} \log\frac{f(x)}{g(x)}\,h(x)\,\text{d}x$$assuming this integral is well-defined [counter-examples are easy to come by].

For instance, if $f$, $g$, and $h$ are densities for the Normal distributions with means $\mu_1$, $\mu_2$, and zero, respectively, all with variance one, the value of $$\int_\mathfrak{X} \log\frac{f(x)}{g(x)}\,h(x)\,\text{d}x$$ is $$\int_\mathfrak{X} \{(x-\mu_1)^2-(x-\mu^2_2)\}\,\varphi(x)\,\text{d}x= \mu_1^2-\mu^2_2\,.$$

Note also that, without the averaging, the product $$\prod_{i=1}^n \frac{f(x_i)}{h(x_i)}$$almost surely converges to zero (when $x_i\sim h(x)\,$). While the product $$\prod_{i=1}^n \frac{f(x_i)}{g(x_i)}$$almost surely converges to zero or infinity depending on whether $g$ or $f$ is closer to $h$ in the Kullback-Leibler divergence sense (when $x_i\sim h(x)\,$).

Xi'an
  • 90,397
  • 9
  • 157
  • 575
  • Could you conclude an answer? Is the last integral non-zero (say when $g=h$)? – Olivier Jan 04 '17 at 15:01
  • 1
    Why should it be zero? If $f=g$ it is zero; if $f=h$ and $g\ne h$ it is positive. And if $g=h$ and $f\ne h$ it is negative. It can also be zero for $f\ne h$, $f\ne g$, and $g \ne h$ if $f$ and $g$ are at an equal distance from $h$. – Xi'an Jan 04 '17 at 15:05
  • What do you mean at equal distance from $h$? Could you elaborate? Your answer is interesting but it doesnt't (yet) directly answer the question. – Olivier Jan 04 '17 at 15:09
  • 1
    The main question. Because $\mathfrak{r}= n \mathfrak{r}_n$, it is the sign of the last integral that determines the asymptotic behaviour of the ratio. – Olivier Jan 04 '17 at 15:17
0

Let $Z_n = \prod^n_i \frac{p(x)}{q(x)}$. Consider the quantity $$W_n = \frac{1}{n}log(Z_n) = \frac{1}{n}\sum_i^n log(\frac{p(x)}{q(x)})$$ By Strong Law of Large Numbers, $$\lim_{n \rightarrow \infty} W_n = E_{q(x)}[log(\frac{p(x)}{q(x)})] = \int_\mathcal{X} log(\frac{p(x)}{q(x)})q(x)dx$$

Since $log(a) < a-1 \ \forall a > 0 $ $ a \neq 1$ and that $\frac{p(x)}{q(x)} > 0$, $p(x) \neq q(x)$

$$W_n \rightarrow \int_\mathcal{X} log(\frac{p(x)}{q(x)})q(x)dx < \int_\mathcal{X} (\frac{p(x)}{q(x)} - 1)q(x)dx = \int_\mathcal{X} p(x)dx - \int_\mathcal{X} q(x)dx = 1 - 1 = 0$$ This gives us $$\lim_{n \rightarrow \infty} W_n < 0 \implies \lim_{n \rightarrow \infty} \frac{1}{n}log(Z_n) < 0 \implies \lim_{n \rightarrow \infty} n \cdot \frac{1}{n}log(Z_n) = -\infty \\ \implies \lim_{n \rightarrow \infty} log(Z_n) = -\infty \\ \implies \lim_{n \rightarrow \infty} Z_n = 0 \ \blacksquare$$

bgao
  • 1