19

Approximate Bayesian computation is a really cool technique for fitting basically any stochastic model, intended for models where the likelihood is intractable (say, you can sample from the model if you fix the parameters but you cannot numerically, algorithmically or analytically calculate the likelihood). When introducing approximate Bayesian computation (ABC) to an audience it is nice to use some example model that is really simple but still somewhat interesting and that has an intractable likelihood.

What would be a good example of a really simple model that still has an intractable likelihood?

Xi'an
  • 90,397
  • 9
  • 157
  • 575
Rasmus Bååth
  • 6,422
  • 34
  • 57
  • 3
    Your socks example is really simple and mostly intractable... – Xi'an Dec 08 '14 at 16:42
  • 2
    Ps: The [socks example link](http://www.sumsar.net/blog/2014/10/tiny-data-and-the-socks-of-karl-broman)... – Xi'an Dec 08 '14 at 16:51
  • Well, I was hoping that the socks would be intractable, but you proved that it wasn't, right? :) – Rasmus Bååth Dec 08 '14 at 16:54
  • 4
    This is a good question! There are various toy examples in the literature but they feel a bit artificial to me. It would be nice to have a really simple model motivated by a real application with an intractable likelihood. I seem to remember seeing David Cox present something along these lines but I haven't seen it published... – Dennis Prangle Dec 09 '14 at 10:18

2 Answers2

14

Two distributions that are used a lot in the literature are:

  • The g-and-k distribution. This is defined by its quantile function (inverse cdf) but has an intractable density. Rayner and MacGillivray (2002) is a good overview of these, and one of many ABC papers which use it as a toy example is Drovandi and Pettitt (2011).
  • Alpha stable distributions. These are defined by their characteristic function but have an intractable density except for a couple of special cases. This has applications in finance and is often used in sequential ABC papers, for example Yildirim et al (2013).
Dennis Prangle
  • 531
  • 2
  • 9
  • 2
    The [g-and-k distribution](http://xianblog.wordpress.com/2011/06/29/quantile-distributions/) is a very good example where the quantile function is simple to express while the likelihood function is not available at all: $$Q(u;A,B,g,k)=A + B\left[1+c\dfrac{1-\exp\{-g\Phi(u)\}}{1+\exp\{-g\Phi(u)\}}\right]\{1+\Phi(u)^2\}^k\Phi(u)$$ The $\alpha$-stable distributions are less simple to explain to newbies. – Xi'an Dec 09 '14 at 10:36
  • 3
    Could someone add examples of situations one would model with these distributions? – conjectures Nov 23 '16 at 16:59
9

One example I came through a few weeks ago and quite like for its simplicity is the following one: given an original normal dataset $$ x_1,\ldots,x_n\stackrel{\text{iid}}{\sim}\text{N}(\theta,\sigma^2)\,, $$ the reported data is (alas!) made of the two-dimensional summary $$ S(x_1,\ldots,x_n)=(\text{med}(x_1,\ldots,x_n),\text{mad}(x_1,\ldots,x_n))\,, $$ which is not sufficient and which does not have a closed form joint density.

Community
  • 1
Xi'an
  • 90,397
  • 9
  • 157
  • 575
  • 3
    Just because the joint density is complicated to write down does not mean it does not have a closed form! "Intractable" is starting to seem like a matter of opinion in this thread. Perhaps you could clear that up by explaining what you mean by "intractable"? – whuber Dec 08 '14 at 17:24
  • 1
    Since I do no of anyone who can compute this density, I call it intractable... Since I have no computer program that can produce the numerical value of this likelihood, I am forced to use an ABC algorithm. – Xi'an Dec 08 '14 at 17:53
  • What I am wondering, though, is if you *do* have an algorithm to compute the likelihood, why should it be considered "intractable"? I can see two useful concepts of "intractable" in this regard: a likelihood is "intractable" if (1) it is so complex that theoretical mathematical analysis is difficult or impossible; or (2) it requires such a computational burden to evaluate accurately as to prohibit its use in practical problems. These are two very different senses of the term--yet neither seems to apply to your example, unless your ABC algorithm is taking a long time to execute. – whuber Dec 08 '14 at 19:00
  • 3
    [ABC](https://arxiv.org/abs/1101.0955) does not compute the likelihood but uses simulations from the data to provide a sample of parameters that is an approximation of the true posterior. At the end of the day/computation, I am not the wiser about the likelihood function and I cannot produce a numerical value for $L(\theta|x_1,\ldots,x_n)$. – Xi'an Dec 08 '14 at 19:36
  • 1
    That suggests yet another interpretation of the question (which very well might be the intended one): "intractable" might mean that finding the posterior is computationally intensive. However, if the focus truly is on finding the posterior, then whether or not you have successfully computed the likelihood would have no bearing on the question, would it? – whuber Dec 08 '14 at 20:20
  • 2
    @whuber If one could successfully compute the likelihood, the example would not be very suitable for demonstrating an algorithm for approximating posteriors _without_ computing likelihood$\times$prior products. – Juho Kokkala Dec 09 '14 at 10:24
  • 4
    @whuber I think your interpretation (2) in the comment beginning "What I am wondering" is at least essentially the intended one. However, I don't understand your last remark "unless your ABC algorithm is taking a long time to execute" - the point of the question is that the expensive likelihood evaluation will be avoided by using ABC instead. – Juho Kokkala Dec 09 '14 at 10:31