4

Notation: let the $\chi^2$-divergence between $p, q$ be defined as

$$\chi^2 (p||q) := \int \left ( \frac{p(x)}{q(x)} \right )^2 q(x)\mathrm{d}x -1 = \int \frac{p(x)}{q(x)} p(x)\mathrm{d}x - 1. $$

Suppose $q$ is a fully known prior distribution, and $p$ is a posterior distribution known up to a normalizing constant but its normalizing constant can be estimated with a reliable Monte Carlo estimator with finite variance. If it helps simplify the problem, assume exact iid samples can be drawn from both distributions.

  1. Which estimator would be less variable? A Monte Carlo estimator with samples from $q$, or samples from $p$?
  2. Suppose I have samples from both $p$ and $q$. Is there a method for leveraging both samples to produce a more stable estimator?
user
  • 2,010
  • 8
  • 23
  • 1
    It depends on the pair $(p,q)$ but it may be that one of the estimators has an infinite variance. – Xi'an Aug 01 '21 at 04:51
  • Reminds me of the problem with Harmonic mean estimator vs a naive Monte Carlo estimator for normalizing constants. – user Aug 01 '21 at 15:12

1 Answers1

3

Given $x_1,\ldots,x_n\sim p(x)$ and $y_1,\ldots,y_m\sim q(x)$, the multiple mixture estimator of Owen and Zhou (2000) is $$\hat\xi=\sum_{i=1}^n \left( \frac{p(x_i)^2}{q(x_i)}\right)\frac{1}{np(x_i)+mq(x_i)}+\sum_{i=1}^m \left( \frac{p(y_i)^2}{q(y_i)}\right)\frac{1}{np(y_i)+mq(y_i)} - 1$$ It is unbiased for $\chi^2(p||q)$ and by mixing both densities it brings stability to the importance weights.

user
  • 2,010
  • 8
  • 23
Xi'an
  • 90,397
  • 9
  • 157
  • 575
  • 1
    I think the expression is missing a $-1$ for the complete $\chi^2$-divergence. Thanks for the answer and reference professor! This method also gave me some idea I want to explore next. – user Aug 01 '21 at 15:10
  • also reminds me of bridge sampling! – user Aug 01 '21 at 17:51