How do ABC and MCMC differ in their applications?

Question

To my understanding Approximate Bayesian Computation (ABC) and Markov Chain Monte Carlo (MCMC) have very similar aims. Below I describe my understanding of these methods and how I perceive the differences in their application to real life data.

Approximate Bayesian Computation

ABC consists at sampling a parameter $\theta$ from a prior, through numerical simulation compute a statistic $x_i$ which is compared to some observed $x_{obs}$. Based on a rejection algorithm, $x_i$ is either retained or rejected. The list of retained $x_i$s made the posterior distribution.

Markov Chain Monte Carlo

MCMC consists at sampling a prior distribution of the parameter $\theta$. It takes a first sample $\theta_1$, compute $P(x_{obs} | \theta_1)P(\theta_1)$ and then jump (according to some rule) to a new value $\theta_2$ for which $P(x_{obs} | \theta_2)P(\theta_2)$ is computed again. The ratio $\frac{P(x_{obs} | \theta_2)P(\theta_2)}{P(x_{obs} | \theta_1)P(\theta_1)}$ is calculated and depending on some threshold value, the next jump will occur from the first or the second position. The exploration of $\theta$ values goes one and one and by the end, the distribution of retained $\theta$ values is the posterior distribution $P(\theta | x)$ (for a reason that is still unknown to me).

I realize that my explanations miss to represent the variety of methods that exists under each of these terms (especially for MCMC).

ABC vs MCMC (pros and cons)

ABC has the advantage that one does not need to be able to analytically solve $P(x | \theta)P(\theta)$. As such ABC is convenient for complex model where MCMC would not make it.

MCMC allows to make statistical tests (likelihood ratio test, G-test, ...) while I don't think this is feasible with ABC.

Am I right so far?

Question

• How do ABC and MCMC differ in their applications? How does one decide to make use of one or another method?

Answer 1

Some additional comments on top of Björn's answer:

1. ABC was first introduced by Rubin (1984) as an explanation of the nature of Bayesian inference, rather than for computational purposes. In this paper he explained how the sampling distribution and the prior distribution interact to produce the posterior distribution.

2. ABC is however primarily exploited for computational reasons. Population geneticists came up with the method on tree-based models where the likelihood of the observed sample was intractable. The MCMC (Data Augmentation) schemes that were available in such settings were awfully inefficient and so was importance sampling, even with a parameter of a single dimension... At its core, ABC is a substitute to Monte Carlo methods like MCMC or PMC when those are not available for all practical purposes. When they are available, ABC appears as a proxy that may be used to calibrate them if it runs faster.

3. In a more modern perspective, I personally consider ABC as an approximate inference method rather than a computational technique. By building an approximate model, one can draw inference on the parameter of interest without necessarily relying on a precise model. While some degree of validation is necessary in this setting, it is not less valid than doing model averaging or non-parametrics. In fact, ABC can be seen as a special type of non-parametric Bayesian statistics.

4. It can also be shown that (noisy) ABC is a perfectly well-defined Bayesian approach if one replaces the original model and data with a noisy one. As such it allows for all Bayesian inferences one can think of. Including testing. Our input to the debate about ABC and hypothesis testing is that the approximate model underlying ABC may end up as poorly equipped to assess the relevance of an hypothesis given the data, but not necessarily, which is just as well since most applications of ABC in population genetics are concerned with model choice.

5. In an even more recent perspective, we can see ABC as a Bayesian version of indirect inference where the parameters of a statistical model are related with the moments of a pre-determined statistic. If this statistic is enough (or sufficient in the vernacular sense) to identify these parameters, ABC can be shown to converge to the true value of the parameters with the number of observations.

Answer 2

The difference is that with ABC you do not need an analytic expression for $P(x|\theta)$ and instead approximate it by simulating data and seeing for which values of $\theta$ simulated data most often (approximately) matches the observed data (with proposed values e.g. drawn randomly from the prior). For simple cases, such as a single binomial random variable with a not too large sample size you can even require an exact match and in those cases, there really is absolutely nothing you could not do with these posterior samples that you could not also do with standard MCMC samples. For more complex situations with continuous (even for multivariate discrete outcomes) and potentially multivariate outcomes requiring an exact match is no longer feasible.

There are in fact MCMC versions of ABC, which addresses the issue that if you have a prior that does not closely resemble the posterior (e.g. because the prior is very uninformative) sampling by drawing from the prior is extremly inefficient, because you very rarely will get a close match between the observed and the simulated data.

When $P(x|\theta)$ is analytically available, I assume it will nearly always be preferrable to use a standard MCMC. I suppose it is conceivable that somehow the evaluation of $P(x|\theta)$ is so incredibly computationally expensive that ABC performs better. Perhaps someone knows an example of this. In contrast I would consider ABC or MCMC-ABC (or one of the many other ABC variants) primarily when a standard MCMC approach is not an option, because $P(x|\theta)$ is not analytically available. Of course there may be some other possible options in such cases (e.g. INLA, quadratic approximations to likelihoods etc.) that may be more efficient/successful for particular problems. In a way, any limitations in what you can do with posterior samples from ABC come from only requiring an aproximate match between the actual and the simulated data (if you could require an exact match, there would be no issues, at all). There are several good introductory papers e.g. this paper by Marin et al. (2012). At least one of the co-authors (@Xi'an) is an active contributor here and I'd love to here his thoughts, too - I believe he may be able to say a lot more on the testing topic.