Exponential family and conjugate priors

Question

Is a distribution that belongs to the exponential family necessarily conjugate prior?

Answer 1

"If the likelihood function belongs to the exponential family, then a conjugate prior exists", not the other way around. For example, many popular discrete distributions come from exponential family and have conjugate priors, but are not conjugate priors for other distributions, because the parameters of most distributions are not discrete.

Answer 2

A bit of reverse engineering: for the likelihood function associated with an exponential family, with natural parameterisation $\theta$ and sufficient statistic $x$ $$\ell(\theta|x)\propto \exp\{\theta^\prime x-\tau(\theta)\}\quad x\in\mathcal X,\ \theta\in\Theta$$ a family of conjugate priors is made of distributions with densities $$\pi(\theta|x_0,\lambda)\propto \exp\{\theta^\prime x_0-\lambda\tau(\theta)\}\quad x_0\in\mathcal X_0,\ \lambda\in\Lambda$$ with respect to an arbitrary dominating measure $\text d\nu$ on $\Theta$, provided

1. $\mathcal X_0+\mathcal X\subset\mathcal X_0$
2. $\Lambda+\mathbb N \subset \Lambda$
3. $\int_\Theta \exp\{\theta^\prime x_0-\lambda\tau(\theta)\}\,\text d\nu(\theta) <\infty$ for all $x_0\in\mathcal X_0,\ \lambda\in\Lambda$.

It clearly defines an exponential family with sufficient statistic $(\theta,\tau(\theta))$.

The reverse problem is to start from an exponential family density on $\theta$ $$\pi(\theta|\xi)\propto\exp\{T(\theta)^\prime\xi-\delta(\xi)\}\propto\exp\{T(\theta)^\prime\xi\}$$ and to identify a likelihood enjoying the above as a conjugate family. This means first breaking $T(\theta)^\prime\xi$ into $\theta^\prime x_0-\lambda\tau(\theta)$ and second finding a measure $\text dμ$ on $\mathcal X$ such that $$\int_{\mathcal X} \exp\{θ′x\}\text dμ(x)=\exp\{τ(θ)\}$$ This is an issue when considering for instance the Poisson distributions $$\pi(\theta|\xi)\propto\exp\{\theta\xi-\log(\theta!)\}\mathbb I_\mathbb N(\theta)$$ because $\theta$ and $\xi$ are unidimensional. One would thus need to embed the Poisson family into a two-dimensional family $$\pi(\theta|\xi,\lambda)\propto \exp\{\theta\xi-\lambda\log(\theta!)\}\mathbb I_\mathbb N(\theta)$$ and then check if there exist $\lambda_0\in\mathbb R$, a set $\mathcal X\subset\mathbb R$, and a measure $\text d\mu$ on $\mathcal X$such that $$\int_\mathcal X \exp\{\theta x\} \,\text d\mu(x)=\exp\{\lambda\log(\theta!)\}\quad \forall\theta\in\mathbb N$$