1

Posteriors for conjugate priors can be analytically derived and save us from doing MCMC.

Conjugate priors simply have a posterior in the same family as the prior distribution. Are there other analytically derivable posteriors that save us from doing MCMC?

Xi'an
  • 90,397
  • 9
  • 157
  • 575
  • 1
    Yes and it saves us from doing MCMC. But, are there any other types of prior-likeliohood combos that save us from MCMC –  Apr 01 '16 at 12:26
  • Distributions that are discrete or continuous mixtures of conjugate priors can also be directly simulated. – Xi'an Apr 06 '16 at 07:15

1 Answers1

2

Anytime the posterior distribution is a known, well studied distribution, chances are we can sample from it. Some prior and likelihood combinations lead to a posterior that is in the same class of distributions as the prior. When this happens, the prior is said to be conjugate for the likelihood.

There are a wide variety of such prior-likelihood combinations like the Normal-Normal, Beta-Binomial etc. A complete list can be found here.

There can be other examples of when the posterior distribution is known, but not necessarily in the same family as the prior.

$$ X|\sigma^2 \sim N(0, \sigma^2) \quad \text{and} \quad \pi(\sigma^2) \propto \dfrac{1}{\sigma^2}.$$

The prior on $\sigma^2$ is improper (and if the parameters in the Inverse-Gamma were allowed to be 0, this would be an Inverse-Gamma distribution, but it is not).

The posterior distribution is $$\sigma^2|X \sim \text{Inverse Gamma}\left(\dfrac{n}{2}, \dfrac{x^2}{2} \right).$$

So, yes there can be other examples of the posterior distribution taking a known form, such that MCMC methods are not needed. In addition, even if the posterior distribution is unknown, it is not necessary that an MCMC procedure is required to sample from the posterior. Rejection sampling may be implemented. For example,

$$X|\theta \sim N(\theta, 1) \quad \text{and} \quad \theta \sim \text{Cauchy}(\mu, \sigma).$$ The resulting posterior is intractable, but a rejection sampler can be implemented to sample from the posterior (I can elaborate on how this is done, but that would be digressing).

Greenparker
  • 14,131
  • 3
  • 36
  • 80
  • Your inverse Gamma example remains a conjugate example in that the prior is an inverse Gamma $Ga(0,0)$ on $\sigma^2$. – Xi'an Apr 06 '16 at 07:14
  • 1
    The Cauchy variable can be written as $\theta=\mu+\sigma \delta \zeta$, where $\zeta\sim\mathcal{N}(0,1)$ and $\delta\sim\mathcal{IG}(1/2,1/2)$. Simulating from the posterior is hence feasible via this mixture representation. – Xi'an Apr 06 '16 at 07:46
  • @Xi'an I was not aware of that mixture representation for the Cauchy distribution. Thanks. Regarding the conjugacy part, this might be contentious, but since the definition of the Inverse Gamma distribution requires the parameters to be positive, $1/\sigma^2$ is not an inverse Gamma distribution. – Greenparker Apr 06 '16 at 10:19
  • 1
    IG(0,0) is the limiting case of a IG(a,b) distribution. While an improper prior, hence not a distribution, the posterior distribution is obtained the same way as with a IG(a,b) prior when a>0,b>0. – Xi'an Apr 06 '16 at 10:34
  • I agree with your comment, but don't think technically "conjugacy" is holds. I could easily be wrong. If you think of a better example, feel free to edit my answer. – Greenparker Apr 06 '16 at 10:52