Suppose I have a parameter (or just any random variable) $\theta$, which we can assume to be absolutely continuous. If I compute a Highest Posterior Probability $I$ for $\theta$ and another one (let's call it $I'$) for $g(\theta)$, will it be true that $I'=g(I)$? That is, are HPD intervals invariant under a reparameterization $g$? Does that change if we assume $g$ to be bijective or perhaps monotonic?
Thanks!
In general, they are not, even under the hypothesis that the transformation is monotonic. This is due to the fact that, when we transform variables, the density of the transformed variable is the determinant of the jacobian multiplied by the density of the original variable. The jacobian "reweights" the density in each point, so points that were in the HPD might not be in the HPD for the transformed variable.
This counter example can be found in the reference. First notice that, if the density is unimodal, then the HPD is an interval. Moreover, except in extreme cases,the endpoints of the interval have the same density value.
Suppose that $\mathbf{x} = (x_i)_{i=1}^n$ is a c.i.i.d. sample with $x_1|\theta \sim exp(\theta)$. Consider the prior $\theta \sim Ga(a,b)$. The posteriori is $\theta|x \sim Ga(A,B)$, with $A = a+n$ and $B = b+\sum_{i=1}^nx_i$. Since $h(\theta|x)$ is unimodal, we have that the HPD with credibility $\gamma$ is given by $(\theta_l, \theta_u)$ that satisfies
$$ h(\theta_l|x) = h(\theta_u|x) \iff \theta_l^{A-1}e^{-B\theta_l} = \theta_u^{A-1}e^{-B\theta_u}\quad.$$
Consider the transformation $\mu = \frac{1}{\theta}$, which is monotonic. We have
$$\gamma = \mathbb{P}(\theta_l < \theta < \theta_u) = \mathbb{P}\left(\frac{1}{\theta_u} < \mu < \frac{1}{\theta_l}\right) \quad.$$
Therefore, the transformed interval has credibility $\gamma$ for $\mu$. Using the Jacobian theorem and some algebra, we can prove that $\mu|x \sim GaI(A,B)$. Hence, the HPD with credibility $\gamma$ for $\mu$ is also an interval. Denote by $q$ the density of $\mu|x$, $K$ being the normalization constant. We have
$$q\left(1/\theta_u|x\right) = K\theta_u^2 \theta_u^{A-1}e^{-B\theta_u} = K\theta_u^2\theta_l^{A-1}e^{-B\theta_l} \neq q\left(1/\theta_l|x\right) \quad.$$
Since the endpoints of the transformed interval do not have the same density, this interval can not be the HPD.
Reference
Paulino, Carlos Daniel Mimoso, Maria Antónia Amaral Turkman, and Bento Murteira. Estatística bayesiana. 2003. 2ª ed. Page 145, Example 3.4.
