4

Given some data $y$, my interest centers around a collection of models $\{\mathcal{M}_1,\mathcal{M}_2,\cdots,\mathcal{M}_L\}$ representing competing hypotheses about $y$. Each model $\mathcal{M}_l$ may be characterized by a model-specific parameter vector $\theta_l$ and sampling density (likelihood) $f(y\,|\,\mathcal{M}_l,\theta_l)$.

where $\pi(\theta_l\,|\,\mathcal{M}_l)$ and $\pi(\theta_l\,|\,y,\mathcal{M}_l)$ are the prior and posterior distributions of $\theta_l$ and $m(y|\mathcal{M}_l)$ is the marginal likelihood of $y$ given $\mathcal{M}_l$. Following Chib (1995) I write $$ m(y|\mathcal{M}_l)=\frac{f(y\,|\,\mathcal{M}_l,\theta_l)\pi(\theta_l\,|\,\mathcal{M}_l)}{\pi(\theta_l\,|\,y,\mathcal{M}_l)} $$

Applying Bayes theorem yet again I can calculate the marginal posterior probability of each model. $$ p(\mathcal{M}_l\,|\,y)=\frac{m(y|\mathcal{M}_l)p(M_l)}{\sum_{i=1}^L m(y|\mathcal{M}_i)p(M_i) } $$

My question is: what methods are best for actually estimating $m(y|\mathcal{M}_l)$ when the posterior distribution is not known? Or similarly when the prior is non-conjugate?

I know of both this estimator $$ m(y|\mathcal{M}_l)=\int_{\Theta_l} f(y\,|\,\mathcal{M}_l,\theta_l)\pi(\theta_l\,|\,\mathcal{M}_l)d\theta_l = E[f(y\,|\,\mathcal{M}_l,\theta_l)\,|\,\mathcal{M}_l] $$ $$\approx \frac{1}{G}\sum_{g=1}^G f(y\,|\,\mathcal{M}_l,\theta^{(g)}_l) $$ Where $\theta^{(1)}_l,\theta^{(2)}_l,\cdots, \theta^{(G)}_l$ are draws from the prior $\pi(\theta_l\,|\,\mathcal{M}_l)$ But Newton and Raftery (1994) says this estimator converges very slowly and recommends the harmonic mean estimator instead $$ \hat{m}(y_t|\mathcal{M}_l)=\bigg[\sum_{g=1}^G \frac{1}{ f(y\,|\,\mathcal{M}_l,\theta^{(g)}_l)} \bigg]^{-1} $$ where parameters are drawn from the posterior. Although consistent, the harmonic mean estimator is noted for being unstable by Chib (1995) and others. My references are 20 years old so I would think researchers have found better methods but I have not had much luck finding them on my own. I was wondering if anyone here knew about good practical means of estimation.

  • Chib, Siddhartha, “Marginal Likelihood from the Gibbs Output,” Journal of the American Statistical Association, 1995, 90 (432), pp. 1313–1321.
  • Newton, Michael A. and Adrian E. Raftery, “Approximate Bayesian Inference with the Weighted Likelihood Bootstrap,” Journal of the Royal Statistical Society. Series B (Methodological), 1994, 56 (1), pp. 3–48.
Zachary Blumenfeld
  • 3,826
  • 1
  • 14
  • 21
  • 2
    We wrote a [comparison study](http://arxiv.org/abs/0910.2325) about those methods with Jean-Michel Marin a few years ago. And yes the Newton-Raftery method should be avoided at all costs! – Xi'an May 24 '15 at 09:27
  • A useful keyword from the physics literature is _thermodynamic integration_, also going under the name of _path sampling_ in statistics. – Xi'an May 24 '15 at 17:26
  • 1
    @Xi'an Thank you for the references I read over both the comparison study you provided and the paper on bridge sampling by Andrew Gelman and Xiao-Li Meng cited therein. It was helpful (though I admittedly don't understand all of it). My situation is a bit particular. I am comparing structural economic models. The likelihoods are based on linear regression models, but the economic theory requires sign restrictions on the parameters. I impose these sign restrictions in my priors, and do not have very much data so it is not easy to choose approximations to the target density. – Zachary Blumenfeld May 25 '15 at 01:57

0 Answers0