Are there any non exponential family distributions with conjugate priors?

Question

I believed I had been taught that only exponential family distributions have conjugate priors but I have recently read that ' all exponential family distributions have conjugate priors', leaving the possibility that there are non exponential family distributions with such. Is this a slip of the tongue or was my original learning mistaken?

Answer 1

The uniform distribution $U(0, \theta)$ has a conjugate prior: the Pareto distribution. Since uniform distributions are not part of the exponential family, this seems like a counter-example.

That said, there is a connection between the exponential family and conjugate priors. The whole point of conjugate priors is to ensure that the prior and posterior have similar forms and, due to the properties of exponents, this is easy to achieve within the exponential family. It also doesn't hurt that most of the commonly-used distributions (normal, poisson, etc) are within the exponential family too.

This chapter by Michael Jordon lays out that argument in more detail.