Does it make sense to condition on fixed values of some parameters before doing MCMC on the other parameters?

Question

I have a Bayesian model with a large number of parameters (around 50), and as usual my goal is to infer the posterior distribution for the parameters, with MCMC.

However, I am only interested in the full posterior distribution for 5 of the parameters - the others are indispensable to the model but uninteresting. Due to the impossible computational cost of th MCMC chain on all 50 parameters, I am looking at this method to give some idea of the posteriors I am interested in, while still being possible to run in the amount of time:

1. Obtain the MAP estimate of all 50 parameters.
2. Obtain the MCMC samples of the posterior distribution for the 5 interesting parameters, conditional on the MAP estimate for the other parameters.

Obviously this conditional posterior is not as good as the correct joint posterior. But as an approximation, do you think the strategy is plausible? Its okay to assume that the parameters using MAP estimates dont have alot of variance.

Answer 1

Since you are interested in simulating $$\pi(\theta_1|\mathbf x) = \int \pi(\theta_1,\theta_2|\mathbf x)\,\text{d}\theta_2$$ you are essentially seeking a manageable approximation to this integral that does not involve simulating the joint $\pi(\theta_1,\theta_2|\mathbf x)$. The MAP proposal is stating that $$\int \pi(\theta_1,\theta_2|\mathbf x)\,\text{d}\theta_2\approx \pi(\theta_1|\hat\theta_2^\text{MAP},\mathbf x)$$ for all $\theta_1$'s which is quite crude. Note that there are two possible choices for the MAP estimate, one being the joint MAP and the other the marginal MAP, presumably impossible to derive.

A less crude version would be to use a Laplace approximation of this integral, replacing $\pi\theta_2|\mathbf x)$ with a Normal centered at the MAP estimate and a variance covariance matrix associated with the Fisher information (or its observed version), $\mathcal N(\hat\theta_2^\text{MAP},\hat\Sigma_2)$. The integral could then be approximated by $$\int \pi(\theta_1,\theta_2|\mathbf x)\,\text{d}\theta_2\approx \frac{1}{N}\sum_{i=1}^N \pi(\theta_1|\theta_2^{(i)},\mathbf x)\qquad\theta_2^{(i)}\sim\mathcal N(\hat\theta_2^\text{MAP},\hat\Sigma_2)$$ A presumably better (and unbiased) approximation is to resort to importance weights $$\int \pi(\theta_1,\theta_2|\mathbf x)\,\text{d}\theta_2\approx \frac{1}{N}\sum_{i=1}^N \frac{\pi(\theta_1|\theta_2^{(i)},\mathbf x)}{\varphi(\theta_2^{(i)}|\hat\theta_2^\text{MAP},\hat\Sigma_2)}\qquad\theta_2^{(i)}\sim\mathcal N(\hat\theta_2^\text{MAP},\hat\Sigma_2)$$ where $\varphi(\theta_2^{(i)}|\hat\theta_2^\text{MAP},\hat\Sigma_2)$ denotes the density of the approximating Normal distribution. A more involved version of this idea is to use integrated nested Laplace approximation (INLA), available in some pseudo-Gaussian settings. (Note that any importance function substitute could be used in the above.)

Note also that Chen, Shao & Ibrahim (1999) have an entire chapter dedicated to the approximation of marginal posterior densities, which may be of help.